Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Le 12/03/2013 13:12, Robert van Leeuwen a écrit : I thought about it, but yet not tried. Which OVS port would you recommend to increase MTU ? On the network node (br-ex or qg-) , or on the compute node (br-int) ? You need to set it on the compute nodes ( int-br-ethX ) and possibly the an extra port on the routing node. (we use a bridge-mapped network to connect to the "outside" world and phy-br-eth1 needs to be set) Cheers, Robert I got it : my issue was isolated on the backend > GRE tunnel TCP response. I then only increased MTU up to 1546 to the physical ethernet device (here, eth0) to allow the 46-byte encap to be not fragmented. Here is the TCPdump stack trace, we can see the GRE headers overhead : 15:01:34.322883 IP (tos 0x0, ttl 64, id 36074, offset 0, flags [DF], proto GRE ( 47), length 1546) 172.16.0.2 > 172.16.0.4: GREv0, Flags [key present], key=0x1, length 1526 IP (tos 0x0, ttl 48, id 26871, offset 0, flags [DF], proto TCP (6), length 1500) X.X.X.X.80 > 10.0.0.4.41507: Flags [P.], cksum 0x6a90 (correct), seq 1420:2880, ack 412, win 3456, length 1460 Here, 172.16.0.2 is the Quantum network node and 172.16.0.4 is the compute node (internal IPs). As the packet was fragmented due to GRE encap, I only changed eth0 on the network node to get things done : ip link set eth0 mtu 1546 Now it works. I assume it's not 100% perfect, being an ugly hack, but that allows to bypass GRE headers overhead in case of PathMTU failing. Thanks all for your help, -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
> I thought about it, but yet not tried. Which OVS port would you > recommend to increase MTU ? > On the network node (br-ex or qg-) , or on the compute node (br-int) ? You need to set it on the compute nodes ( int-br-ethX ) and possibly the an extra port on the routing node. (we use a bridge-mapped network to connect to the "outside" world and phy-br-eth1 needs to be set) Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Le 12/03/2013 09:20, Robert van Leeuwen a écrit : Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). I think this is the same issue: https://bugs.launchpad.net/quantum/+bug/1075336 So instead of decreasing the MTU on the "physical interface" you could also increase it on the openvswitch port. Cheers, Robert I thought about it, but yet not tried. Which OVS port would you recommend to increase MTU ? On the network node (br-ex or qg-) , or on the compute node (br-int) ? Thanks, -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
> Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). I think this is the same issue: https://bugs.launchpad.net/quantum/+bug/1075336 So instead of decreasing the MTU on the "physical interface" you could also increase it on the openvswitch port. Cheers, Robert ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
On 03/11/2013 06:09 AM, Sylvain Bauza wrote: Okay. I think I got the reason why it's not working with OVS/GRE contrary to FlatDHCP nova-network. So, as per http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml , GRE encapsulation protocol can add up to 34 bytes to the IP datagram (meaning the TCP segment is only 1456 bytes if MTU set to 1500). When the packet is about 1500 bytes, then it should fragment to keep the 1500-byte size of the reply (including GRE encap then). That sounds like the reason. Unfortunaly, due to security purpose, the ICMP packet "type 3/code 4" (frag. needed) can't be reached to the X.X.X.X backend as this backend is denying any ICMP request (firewall). As a consequence, PathMTU is failing and packets still retransmited with 1500-byte size again and again... As said on my first post, the only workaround I found is to modify *all* my VMs with MTU set to 1454 (don't know why there is a 2-bytes overhead compared to the 1456-byte I told above), including my Windows VMs which is not a cool stuff (modifying a registry key and reboot the VM. Yes, you aren't dreaming. This is the way for Windows-based machines to modify MTUs...) Do you know if any cool idea would prevent to modify VMs, and only do things on the network node ? Yes. Let the ICMP Destination Unreachable, Datagram Too Big messages through. So the network can function the way it was intended. Otherwise you have no recourse but alter the MTU in the VMs. Or add the insult of tweaking the code to ignore the DF bit to the injury of blocking the ICMP messages. (Assuming that is even possible) If you are Very Lucky (tm) all your network infrastructure in the broadcast domain (everything on the same side of a router - device forwarding based on Layer3 (eg IP) addressing or put another way, everything reachable via just switches - in the proper sense of the term wherein a switch is a device making forwarding decisions based on layer2, eg Ethernet addresses) then you can try to increase the MTU of your physical interfaces so the GRE encapsulation overhead can be "hidden" from the VMs. But *everything* in the broadcast domain must have the same maximum frame size (MTU) or life becomes even more "interesting." My suggestion is let the ICMP Destination Unreachable, Datagram Too Big messages through. It is perhaps my failing, but I fail to see how blocking them improves "security." rick jones adde parvum parvo magnus acervus erit - Ovid quoted in The Mythical Man Month ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Okay. I think I got the reason why it's not working with OVS/GRE contrary to FlatDHCP nova-network. So, as per http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml , GRE encapsulation protocol can add up to 34 bytes to the IP datagram (meaning the TCP segment is only 1456 bytes if MTU set to 1500). When the packet is about 1500 bytes, then it should fragment to keep the 1500-byte size of the reply (including GRE encap then). Unfortunaly, due to security purpose, the ICMP packet "type 3/code 4" (frag. needed) can't be reached to the X.X.X.X backend as this backend is denying any ICMP request (firewall). As a consequence, PathMTU is failing and packets still retransmited with 1500-byte size again and again... As said on my first post, the only workaround I found is to modify *all* my VMs with MTU set to 1454 (don't know why there is a 2-bytes overhead compared to the 1456-byte I told above), including my Windows VMs which is not a cool stuff (modifying a registry key and reboot the VM. Yes, you aren't dreaming. This is the way for Windows-based machines to modify MTUs...) Do you know if any cool idea would prevent to modify VMs, and only do things on the network node ? My TCP/IP knowledge is quite at its limits, so any idea is great for me. Thanks, -Sylvain (BTW, maybe my explanation is absolutely wrong, and GRE is not responsible of the 36-byte overhead. If yes, please accept my apologies, any other clarification would be great). Le 11/03/2013 10:07, Sylvain Bauza a écrit : I also forgot to mention: I'm using a typical Openvswitch setup with GRE encapsulation. I can't proof, but would GRE not able to work with PathMTU ? -Sylvain Le 11/03/2013 09:40, Sylvain Bauza a écrit : Hi Rick, reply inline. Le 08/03/2013 20:27, Rick Jones a écrit : On 03/08/2013 09:55 AM, Aaron Rosen wrote: Hi Sylvain, This seems very odd to me. The reason this should happen is if your client is sending packets with the DF (don't fragment) bit set in the TCP header of the packets you are sending. I'd confirm that your version of 'curl' is doing this (which it should definitely not do!). Why shouldn't a TCP connection initiated by curl (or anything else) have Path MTU discovery enabled? (ie the DF bit set in the IP datagrams carrying the TCP segments) [SBA] Thanks for the explanation of the DF flag What should happen is the router should fragment the packets for you and if a fragment is lost TCP will just re-transmit the full packet again and things should eventually work Here I thought all the IETF demigods considered IP Fragmentation 'To Be Avoided (tm)' - hence the creation of Path MTU discovery in the first place. :) FWIW, in the IPv6 world, routers do not fragment. That implies either functioning PathMTU discovery, or lowest common MTU... Aaron On Fri, Mar 8, 2013 at 9:08 AM, Sylvain Bauza wrote: Hi, I recently observed a strange behaviour with L3 Quantum routing (Openvswitch setup with Provider Router). A simple curl to an external website is sometimes failing due to packet size : 192.168.10.3 > X.X.X.X: ICMP 192.168.10.3 unreachable - need to frag (mtu 1454), length 556 IP (tos 0x0, ttl 48, id 25918, offset 0, flags [DF], proto TCP (6), length 1500) Why is the ICMP Destination Unreachable datagram being sent back so large? I would have expected it to be rather smaller - an Ethernet, IP and ICMP header, and then the original IP header and something like 8 bytes or so of the original IP datagram's payload. I take it that ICMP is not getting back to the original sender? Or is being ignored? [SBA] I take the point. That means that PathMTU is not working for my Quantum installation. I also had a Nova-network (FlatDHCP mode) and I didn't noticed the issue. So, I assume something is wrong with my config. Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). For info, 192.168.10.3 is the floating IP bound to 10.0.0.4 (private IP). I suppose if 10.0.0.4 doesn't explicitly know about 192.168.10.3 it might indeed ignore the ICMP message. Assuming it isn't getting un-NATted on the way back. [SBA] This *is* un-NAT'd on the way back. By tcpdump'ing with the '-i any' interface, I can see the DNAT mapping on the way back : Do you have any idea on what I should fix (or at least workaround) to have PathMTU working ? By the way, I did check and both client (10.0.0.4) and server (X.X.X.X) have MTU set to 1500. I can't understand why the server is asking for a fragment size of 1454. Thanks, -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
I also forgot to mention: I'm using a typical Openvswitch setup with GRE encapsulation. I can't proof, but would GRE not able to work with PathMTU ? -Sylvain Le 11/03/2013 09:40, Sylvain Bauza a écrit : Hi Rick, reply inline. Le 08/03/2013 20:27, Rick Jones a écrit : On 03/08/2013 09:55 AM, Aaron Rosen wrote: Hi Sylvain, This seems very odd to me. The reason this should happen is if your client is sending packets with the DF (don't fragment) bit set in the TCP header of the packets you are sending. I'd confirm that your version of 'curl' is doing this (which it should definitely not do!). Why shouldn't a TCP connection initiated by curl (or anything else) have Path MTU discovery enabled? (ie the DF bit set in the IP datagrams carrying the TCP segments) [SBA] Thanks for the explanation of the DF flag What should happen is the router should fragment the packets for you and if a fragment is lost TCP will just re-transmit the full packet again and things should eventually work Here I thought all the IETF demigods considered IP Fragmentation 'To Be Avoided (tm)' - hence the creation of Path MTU discovery in the first place. :) FWIW, in the IPv6 world, routers do not fragment. That implies either functioning PathMTU discovery, or lowest common MTU... Aaron On Fri, Mar 8, 2013 at 9:08 AM, Sylvain Bauza wrote: Hi, I recently observed a strange behaviour with L3 Quantum routing (Openvswitch setup with Provider Router). A simple curl to an external website is sometimes failing due to packet size : 192.168.10.3 > X.X.X.X: ICMP 192.168.10.3 unreachable - need to frag (mtu 1454), length 556 IP (tos 0x0, ttl 48, id 25918, offset 0, flags [DF], proto TCP (6), length 1500) Why is the ICMP Destination Unreachable datagram being sent back so large? I would have expected it to be rather smaller - an Ethernet, IP and ICMP header, and then the original IP header and something like 8 bytes or so of the original IP datagram's payload. I take it that ICMP is not getting back to the original sender? Or is being ignored? [SBA] I take the point. That means that PathMTU is not working for my Quantum installation. I also had a Nova-network (FlatDHCP mode) and I didn't noticed the issue. So, I assume something is wrong with my config. Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). For info, 192.168.10.3 is the floating IP bound to 10.0.0.4 (private IP). I suppose if 10.0.0.4 doesn't explicitly know about 192.168.10.3 it might indeed ignore the ICMP message. Assuming it isn't getting un-NATted on the way back. [SBA] This *is* un-NAT'd on the way back. By tcpdump'ing with the '-i any' interface, I can see the DNAT mapping on the way back : Do you have any idea on what I should fix (or at least workaround) to have PathMTU working ? By the way, I did check and both client (10.0.0.4) and server (X.X.X.X) have MTU set to 1500. I can't understand why the server is asking for a fragment size of 1454. Thanks, -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Hi Rick, reply inline. Le 08/03/2013 20:27, Rick Jones a écrit : On 03/08/2013 09:55 AM, Aaron Rosen wrote: Hi Sylvain, This seems very odd to me. The reason this should happen is if your client is sending packets with the DF (don't fragment) bit set in the TCP header of the packets you are sending. I'd confirm that your version of 'curl' is doing this (which it should definitely not do!). Why shouldn't a TCP connection initiated by curl (or anything else) have Path MTU discovery enabled? (ie the DF bit set in the IP datagrams carrying the TCP segments) [SBA] Thanks for the explanation of the DF flag What should happen is the router should fragment the packets for you and if a fragment is lost TCP will just re-transmit the full packet again and things should eventually work Here I thought all the IETF demigods considered IP Fragmentation 'To Be Avoided (tm)' - hence the creation of Path MTU discovery in the first place. :) FWIW, in the IPv6 world, routers do not fragment. That implies either functioning PathMTU discovery, or lowest common MTU... Aaron On Fri, Mar 8, 2013 at 9:08 AM, Sylvain Bauza wrote: Hi, I recently observed a strange behaviour with L3 Quantum routing (Openvswitch setup with Provider Router). A simple curl to an external website is sometimes failing due to packet size : 192.168.10.3 > X.X.X.X: ICMP 192.168.10.3 unreachable - need to frag (mtu 1454), length 556 IP (tos 0x0, ttl 48, id 25918, offset 0, flags [DF], proto TCP (6), length 1500) Why is the ICMP Destination Unreachable datagram being sent back so large? I would have expected it to be rather smaller - an Ethernet, IP and ICMP header, and then the original IP header and something like 8 bytes or so of the original IP datagram's payload. I take it that ICMP is not getting back to the original sender? Or is being ignored? [SBA] I take the point. That means that PathMTU is not working for my Quantum installation. I also had a Nova-network (FlatDHCP mode) and I didn't noticed the issue. So, I assume something is wrong with my config. Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). For info, 192.168.10.3 is the floating IP bound to 10.0.0.4 (private IP). I suppose if 10.0.0.4 doesn't explicitly know about 192.168.10.3 it might indeed ignore the ICMP message. Assuming it isn't getting un-NATted on the way back. [SBA] This *is* un-NAT'd on the way back. By tcpdump'ing with the '-i any' interface, I can see the DNAT mapping on the way back : Do you have any idea on what I should fix (or at least workaround) to have PathMTU working ? By the way, I did check and both client (10.0.0.4) and server (X.X.X.X) have MTU set to 1500. I can't understand why the server is asking for a fragment size of 1454. Thanks, -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Thanks Rick that makes sense. I've never run into this myself. I just assumed that a host would by default just set packets at it's MTU size and the network would handle the fragmentation for you. Good to know. Aaron On Fri, Mar 8, 2013 at 11:56 AM, Rick Jones wrote: > On 03/08/2013 11:49 AM, Aaron Rosen wrote: >> >> Hi Rick, >> >> You are right. I just ran curl to test for myself and it does set the >> DF bit. Why is this? Any ideas why it specifies that the packet cannot >> be fragmented? > > > Because most, if not virtually all TCP stacks going back to the mid 1990s > (RFC 1191 is from 1990 and I figured a couple years to propagate) enable > Path MTU discovery by default for TCP. At least those with which I have > come into contact. > > I doubt that curl itself asked for it. I suspect you will find the DF bit > set in the IP datagrams carrying the TCP segments of any application on your > system using TCP - even netperf :) PathMTU discovery, for TCP at least, and > perhaps other "reliable" transports, is considered a Best Practice (tm) and > so enabled by default. Where it may not be enabled by default is for UDP. > > rick ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
On 03/08/2013 11:49 AM, Aaron Rosen wrote: Hi Rick, You are right. I just ran curl to test for myself and it does set the DF bit. Why is this? Any ideas why it specifies that the packet cannot be fragmented? Because most, if not virtually all TCP stacks going back to the mid 1990s (RFC 1191 is from 1990 and I figured a couple years to propagate) enable Path MTU discovery by default for TCP. At least those with which I have come into contact. I doubt that curl itself asked for it. I suspect you will find the DF bit set in the IP datagrams carrying the TCP segments of any application on your system using TCP - even netperf :) PathMTU discovery, for TCP at least, and perhaps other "reliable" transports, is considered a Best Practice (tm) and so enabled by default. Where it may not be enabled by default is for UDP. rick ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
On Mar 8, 2013, at 1:54 PM, Brad Knowles wrote: > On Mar 8, 2013, at 1:49 PM, Aaron Rosen wrote: > >> You are right. I just ran curl to test for myself and it does set the >> DF bit. Why is this? Any ideas why it specifies that the packet cannot >> be fragmented? > > http://lmgtfy.com/?q=don%27t+fragment+bit+path+mtu Sorry, that URL didn't cut-n-paste correctly. Try this instead: http://lmgtfy.com/?q=don%27t+fragment+bit+path+mtu -- Brad Knowles Senior Consultant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
On Mar 8, 2013, at 1:49 PM, Aaron Rosen wrote: > You are right. I just ran curl to test for myself and it does set the > DF bit. Why is this? Any ideas why it specifies that the packet cannot > be fragmented? http://lmgtfy.com/?q=don%27t+fragment+bit+path+mtu -- Brad Knowles Senior Consultant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Hi Rick, You are right. I just ran curl to test for myself and it does set the DF bit. Why is this? Any ideas why it specifies that the packet cannot be fragmented? Thanks, Aaron On Fri, Mar 8, 2013 at 11:27 AM, Rick Jones wrote: > On 03/08/2013 09:55 AM, Aaron Rosen wrote: >> >> Hi Sylvain, >> >> >> This seems very odd to me. The reason this should happen is if your >> client is sending packets with the DF (don't fragment) bit set in the >> TCP header of the packets you are sending. I'd confirm that your >> version of 'curl' is doing this (which it should definitely not do!). > > > Why shouldn't a TCP connection initiated by curl (or anything else) have > Path MTU discovery enabled? (ie the DF bit set in the IP datagrams carrying > the TCP segments) > > >> What should happen is the router should fragment the packets for you >> and if a fragment is lost TCP will just re-transmit the full packet >> again and things should eventually work > > > Here I thought all the IETF demigods considered IP Fragmentation 'To Be > Avoided (tm)' - hence the creation of Path MTU discovery in the first place. > :) > > FWIW, in the IPv6 world, routers do not fragment. That implies either > functioning PathMTU discovery, or lowest common MTU... > > >> >> Aaron >> >> >> On Fri, Mar 8, 2013 at 9:08 AM, Sylvain Bauza >> wrote: >>> >>> Hi, >>> >>> I recently observed a strange behaviour with L3 Quantum routing >>> (Openvswitch >>> setup with Provider Router). A simple curl to an external website is >>> sometimes failing due to packet size : >>> >>> 192.168.10.3 > X.X.X.X: ICMP 192.168.10.3 unreachable - need to frag >>> (mtu 1454), length 556 >>> IP (tos 0x0, ttl 48, id 25918, offset 0, flags [DF], proto TCP (6), >>> length 1500) > > > Why is the ICMP Destination Unreachable datagram being sent back so large? > I would have expected it to be rather smaller - an Ethernet, IP and ICMP > header, and then the original IP header and something like 8 bytes or so of > the original IP datagram's payload. > > I take it that ICMP is not getting back to the original sender? Or is being > ignored? > > > >>> >>> Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu >>> 1454'). >>> >>> For info, 192.168.10.3 is the floating IP bound to 10.0.0.4 (private IP). > > > I suppose if 10.0.0.4 doesn't explicitly know about 192.168.10.3 it might > indeed ignore the ICMP message. Assuming it isn't getting un-NATted on the > way back. > > rick jones > > >>> >>> I can't provide the URL for reproducing, as the external website is >>> actually >>> an external corporate webservice. >>> Do you have any idea on what could be the root cause, and how to fix it ? >>> >>> Thanks, >>> -Sylvain >>> >>> >>> >>> ___ >>> Mailing list: https://launchpad.net/~openstack >>> Post to : openstack@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~openstack >>> More help : https://help.launchpad.net/ListHelp >> >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
On 03/08/2013 09:55 AM, Aaron Rosen wrote: Hi Sylvain, This seems very odd to me. The reason this should happen is if your client is sending packets with the DF (don't fragment) bit set in the TCP header of the packets you are sending. I'd confirm that your version of 'curl' is doing this (which it should definitely not do!). Why shouldn't a TCP connection initiated by curl (or anything else) have Path MTU discovery enabled? (ie the DF bit set in the IP datagrams carrying the TCP segments) What should happen is the router should fragment the packets for you and if a fragment is lost TCP will just re-transmit the full packet again and things should eventually work Here I thought all the IETF demigods considered IP Fragmentation 'To Be Avoided (tm)' - hence the creation of Path MTU discovery in the first place. :) FWIW, in the IPv6 world, routers do not fragment. That implies either functioning PathMTU discovery, or lowest common MTU... Aaron On Fri, Mar 8, 2013 at 9:08 AM, Sylvain Bauza wrote: Hi, I recently observed a strange behaviour with L3 Quantum routing (Openvswitch setup with Provider Router). A simple curl to an external website is sometimes failing due to packet size : 192.168.10.3 > X.X.X.X: ICMP 192.168.10.3 unreachable - need to frag (mtu 1454), length 556 IP (tos 0x0, ttl 48, id 25918, offset 0, flags [DF], proto TCP (6), length 1500) Why is the ICMP Destination Unreachable datagram being sent back so large? I would have expected it to be rather smaller - an Ethernet, IP and ICMP header, and then the original IP header and something like 8 bytes or so of the original IP datagram's payload. I take it that ICMP is not getting back to the original sender? Or is being ignored? Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). For info, 192.168.10.3 is the floating IP bound to 10.0.0.4 (private IP). I suppose if 10.0.0.4 doesn't explicitly know about 192.168.10.3 it might indeed ignore the ICMP message. Assuming it isn't getting un-NATted on the way back. rick jones I can't provide the URL for reproducing, as the external website is actually an external corporate webservice. Do you have any idea on what could be the root cause, and how to fix it ? Thanks, -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Hi Sylvain, This seems very odd to me. The reason this should happen is if your client is sending packets with the DF (don't fragment) bit set in the TCP header of the packets you are sending. I'd confirm that your version of 'curl' is doing this (which it should definitely not do!). What should happen is the router should fragment the packets for you and if a fragment is lost TCP will just re-transmit the full packet again and things should eventually work Aaron On Fri, Mar 8, 2013 at 9:08 AM, Sylvain Bauza wrote: > Hi, > > I recently observed a strange behaviour with L3 Quantum routing (Openvswitch > setup with Provider Router). A simple curl to an external website is > sometimes failing due to packet size : > > 192.168.10.3 > X.X.X.X: ICMP 192.168.10.3 unreachable - need to frag > (mtu 1454), length 556 > IP (tos 0x0, ttl 48, id 25918, offset 0, flags [DF], proto TCP (6), > length 1500) > > Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). > > For info, 192.168.10.3 is the floating IP bound to 10.0.0.4 (private IP). > > I can't provide the URL for reproducing, as the external website is actually > an external corporate webservice. > Do you have any idea on what could be the root cause, and how to fix it ? > > Thanks, > -Sylvain > > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [QUANTUM] (Bug ?) L3 routing not correctly fragmenting packets ?
Hi, I recently observed a strange behaviour with L3 Quantum routing (Openvswitch setup with Provider Router). A simple curl to an external website is sometimes failing due to packet size : 192.168.10.3 > X.X.X.X: ICMP 192.168.10.3 unreachable - need to frag (mtu 1454), length 556 IP (tos 0x0, ttl 48, id 25918, offset 0, flags [DF], proto TCP (6), length 1500) Only changing the VM MTU to 1454 does the trick ('ifconfig eth0 mtu 1454'). For info, 192.168.10.3 is the floating IP bound to 10.0.0.4 (private IP). I can't provide the URL for reproducing, as the external website is actually an external corporate webservice. Do you have any idea on what could be the root cause, and how to fix it ? Thanks, -Sylvain ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp