Re: [openstack-dev] [Neutron] MTU configuration pain

On Mon, Jan 25, 2016 at 08:16:03PM EST, Fox, Kevin M wrote: > Another place to look... > I've had to use network_device_mtu=9000 in nova's config as well to get mtu's > working smoothly. > I'll have to read the code on the Nova side and familiarize myself, but this sounds like a case of DRY

Re: [openstack-dev] [Neutron] MTU configuration pain

big +1 from me. :) Kevin From: Sean M. Collins [s...@coreitpro.com] Sent: Tuesday, January 26, 2016 9:59 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Neutron] MTU configuration pain On Mon, Jan 25, 2016

Re: [openstack-dev] [Neutron] MTU configuration pain

As I recall, network_device_mtu sets up the MTU on a bunch of structures independently of whatever the correct value is. It was a bit of a workaround back in the day and is still a bit of a workaround now. I'd sooner we actually fix up the new mechanism (which is kind of hard to do when the

Re: [openstack-dev] [Neutron] MTU configuration pain

On Mon, Jan 25, 2016 at 01:37:55AM EST, Kevin Benton wrote: > At a minimum I think we should pick a default in devstack and dump a > warning in neutron if operators don't specify it. Here's the DevStack change that implements this. https://review.openstack.org/#/c/267604/ Again this just fixes

Re: [openstack-dev] [Neutron] MTU configuration pain

BTW, regarding devstack: See https://bugs.launchpad.net/devstack/+bug/1532924. I have been trying to get the current code to work, following the ideas in https://specs.openstack.org/openstack/fuel-specs/specs/7.0/jumbo-frames-between-instances.html#proposed-change . It fails only at the last

Re: [openstack-dev] [Neutron] MTU configuration pain

Ian, Overthinking and corner cases led to the existing implementation which doesn't solve the MTU problem and arguably makes the situation worse because options in the configuration files give operators the impression they can control it. For example, the segment_mtu does nothing in the in-tree

Re: [openstack-dev] [Neutron] MTU configuration pain

You need to set path_mtu. https://review.openstack.org/#/c/267604/ sets it now and defaults to 1500 - then Neutron calculates the overhead for your tunnel protocol down to the appropriate value -- Sean M. Collins __

Re: [openstack-dev] [Neutron] MTU configuration pain

On 01/24/2016 07:43 PM, Ian Wells wrote: Also, I say 9000, but why is 9000 even the right number? While that may have been a rhetorical question... Because that is the value Alteon picked in the late 1990s when they created the de facto standard for "Jumbo Frames" by including it in their

Re: [openstack-dev] [Neutron] MTU configuration pain

On 25 January 2016 at 07:06, Matt Kassawara wrote: > Overthinking and corner cases led to the existing implementation which doesn't solve the MTU problem and arguably makes the situation worse because options in the configuration files give operators the impression they can

Re: [openstack-dev] [Neutron] MTU configuration pain

(not for usage questions) Subject: Re: [openstack-dev] [Neutron] MTU configuration pain Results from the Open vSwitch agent... I highly recommend reading further, but here's the TL;DR: Using physical network interfaces with MTUs larger than 1500 reveals problems in several places, but only involving

Re: [openstack-dev] [Neutron] MTU configuration pain

Results from the Open vSwitch agent... I highly recommend reading further, but here's the TL;DR: Using physical network interfaces with MTUs larger than 1500 reveals problems in several places, but only involving Linux components rather than Open vSwitch components (such as br-int) on both the

Re: [openstack-dev] [Neutron] MTU configuration pain

On 23 January 2016 at 11:27, Adam Lawson wrote: > For the sake of over-simplification, is there ever a reason to NOT enable > jumbo frames in a cloud/SDN context where most of the traffic is between > virtual elements that all support it? I understand that some switches do >

Re: [openstack-dev] [Neutron] MTU configuration pain

I believe the issue is that the default is unspecified, which leads to nothing being advertised to VMs via dhcp/ra. So VMs end up using 1500, which leads to a catastrophe when running on an overlay on a 1500 underlay. On Jan 24, 2016 20:48, "Ian Wells" wrote: > On 23

Re: [openstack-dev] [Neutron] MTU configuration pain

I wrote the spec for the MTU work that's in the Neutron API today. It haunts my nightmares. I learned so many nasty corner cases for MTU, and you're treading that same dark path. I'd first like to point out a few things that change the implications of what you're reporting in strange ways. [1]

Re: [openstack-dev] [Neutron] MTU configuration pain

At a minimum I think we should pick a default in devstack and dump a warning in neutron if operators don't specify it. I would still be preferable to changing the default even though it's a behavior change considering the current behavior is annoying. :) On Jan 24, 2016 23:31, "Ian Wells"

Re: [openstack-dev] [Neutron] MTU configuration pain

On 24 January 2016 at 20:18, Kevin Benton wrote: > I believe the issue is that the default is unspecified, which leads to > nothing being advertised to VMs via dhcp/ra. So VMs end up using 1500, > which leads to a catastrophe when running on an overlay on a 1500 underlay. >

Re: [openstack-dev] [Neutron] MTU configuration pain

>The reason for that was in the other half of the thread - it's not possible to magically discover these things from within Openstack's own code because the relevant settings span more than just one server IMO it's better to have a default of 1500 rather than let VMs automatically default to 1500

Re: [openstack-dev] [Neutron] MTU configuration pain

I like both of those ideas. On 24 January 2016 at 22:37, Kevin Benton wrote: > At a minimum I think we should pick a default in devstack and dump a > warning in neutron if operators don't specify it. > > I would still be preferable to changing the default even though it's a >

Re: [openstack-dev] [Neutron] MTU configuration pain

On 24 January 2016 at 22:12, Kevin Benton wrote: > >The reason for that was in the other half of the thread - it's not > possible to magically discover these things from within Openstack's own > code because the relevant settings span more than just one server > > IMO it's

Re: [openstack-dev] [Neutron] MTU configuration pain

Actually, I note that that document is Juno and there doesn't seem to be anything at all in the Liberty guide now, so the answer is probably to add settings for path_mtu and segment_mtu in the recommended Neutron configuration. On 24 January 2016 at 22:26, Ian Wells

Re: [openstack-dev] [Neutron] MTU configuration pain

For the sake of over-simplification, is there ever a reason to NOT enable jumbo frames in a cloud/SDN context where most of the traffic is between virtual elements that all support it? I understand that some switches do not support it and traffic form the web doesn't support it either but besides

Re: [openstack-dev] [Neutron] MTU configuration pain

Adam Lawson wrote on 01/23/2016 02:27:46 PM: > For the sake of over-simplification, is there ever a reason to NOT > enable jumbo frames in a cloud/SDN context where most of the traffic > is between virtual elements that all support it? I understand that > some switches do

Re: [openstack-dev] [Neutron] MTU configuration pain

Adam, Any modern datacenter network, especially those with 10 Gbps or faster connectivity, should support jumbo frames for performance reasons. However, depending on the network infrastructure, jumbo frames does not always mean a 9000 MTU, so neutron should support a configurable value rather

Re: [openstack-dev] [Neutron] MTU configuration pain

The fun continues, now using an OpenStack deployment on physical hardware that supports jumbo frames with 9000 MTU and IPv4/IPv6. This experiment still uses Linux bridge for consistency. I'm planning to run similar experiments with Open vSwitch and Open Virtual Network (OVN) in the next week. I

Re: [openstack-dev] [Neutron] MTU configuration pain

On Tue, Jan 19, 2016 at 08:15:18AM EST, Matt Kassawara wrote: > No. However, we ought to determine what happens when both DHCP and RA > advertise it. We'd have to look at the RFCs for how hosts are supposed to behave since IPv6 has a minimum MTU of 1280 bytes while IPv4's minimum mtu is 576 (what

Re: [openstack-dev] [Neutron] MTU configuration pain

On 01/20/2016 08:56 AM, Sean M. Collins wrote: On Tue, Jan 19, 2016 at 08:15:18AM EST, Matt Kassawara wrote: No. However, we ought to determine what happens when both DHCP and RA advertise it. We'd have to look at the RFCs for how hosts are supposed to behave since IPv6 has a minimum MTU of

Re: [openstack-dev] [Neutron] MTU configuration pain

Maybe it will do the obvious thing and add them together. ;) On Jan 20, 2016 12:03, "Sean M. Collins" wrote: > On Tue, Jan 19, 2016 at 08:15:18AM EST, Matt Kassawara wrote: > > No. However, we ought to determine what happens when both DHCP and RA > > advertise it. > > We'd

Re: [openstack-dev] [Neutron] MTU configuration pain

No. However, we ought to determine what happens when both DHCP and RA advertise it. On Tue, Jan 19, 2016 at 12:36 AM, Kevin Benton wrote: > >Yup. We mostly attempt to do that now. > > Right, but not by default. Can you think of a scenario where advertising > it would be

Re: [openstack-dev] [Neutron] MTU configuration pain

On Mon, Jan 18, 2016 at 4:14 PM, Kevin Benton wrote: > Thanks for the awesome writeup. > > >5) A bridge or veth pair with an IP address can participate in path MTU > discovery (PMTUD). However, these devices do not appear to understand > namespaces and originate the ICMP

Re: [openstack-dev] [Neutron] MTU configuration pain

>Yup. We mostly attempt to do that now. Right, but not by default. Can you think of a scenario where advertising it would be harmful? On Jan 18, 2016 23:57, "Matt Kassawara" wrote: > > > On Mon, Jan 18, 2016 at 4:14 PM, Kevin Benton wrote: > >> Thanks

Re: [openstack-dev] [Neutron] MTU configuration pain

On Sun, Jan 17, 2016 at 8:30 PM, Matt Kassawara wrote: > Prior attempts to solve the MTU problem in neutron simply band-aid it or > become too complex from feature creep or edge cases that mask the primary > goal of a simple implementation that works for most deployments.

Re: [openstack-dev] [Neutron] MTU configuration pain

Thanks for the awesome writeup. >5) A bridge or veth pair with an IP address can participate in path MTU discovery (PMTUD). However, these devices do not appear to understand namespaces and originate the ICMP message from the host instead of a namespace. Therefore, the message never reaches the

Re: [openstack-dev] [Neutron] MTU configuration pain

The MTU setting is an issue because it involves knowledge of the network outside of openstack. That's why it was just a config value that was expected to be set by an operator. This thread is working to see if we can figure that out, or maybe at least come up with a different sub-optimal default.

[openstack-dev] [Neutron] MTU configuration pain

MTU has been an ongoing issue in Neutron for _years_. It's such a hassle, that most people just throw up their hands and set their physical infrastructure to jumbo frames. We even document it. http://docs.openstack.org/juno/install-guide/install/apt-debian/content/neutron-network-node.html >