Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Peters, Rawlin]

On Wednesday, June 15, 2016 12:45 PM, Mooney, Sean K [sean.k.moo...@intel.com] 
wrote:
> > On Tuesday, June 14, 2016 6:27 PM, Kevin Benton (ke...@benton.pub)
> > wrote:
> > > >which generates an arbitrary name
> > >
> > > I'm not a fan of this approach because it requires coordinated
> > assumptions.
> > > With the OVS hybrid plug strategy we have to make guesses on the
> > > agent side about the presence of bridges with specific names that we
> > > never explicitly requested and that we were never explicitly told
> > > about. So we end up with code like [1] that is looking for a
> > > particular end of a veth pair it just hopes is there so the rules have an
> effect.
> [Mooney, Sean K] I really would like to avoid encoding knowledge to
> Generate the names the same way in both neutron and os-vf/nova or having
> any Other special casing to figure out the bridge or interface names.
> 
> >
> > I don't think this should be viewed as a downside of Strategy 1
> > because, at least when we use patch port pairs, we can easily get the
> > peer name from the port on br-int, then use the equivalent of
> > "ovs-vsctl iface-to- br "
> > to get the name of the bridge. If we allow supporting veth pairs to
> > implement the subports, then getting the arbitrary trunk bridge/veth
> > names isn't as trivial.
> >
> > This also brings up the question: do we even need to support veth
> > pairs over patch port pairs anymore? Are there any distros out there
> > that support openstack but not OVS patch ports?
> [Mooney, Sean K] that is a separate discussions In general im in favor of
> deprecating support for veth interconnect with ovs And removing it in ocata.
> I belive I was originally added in juno for centos and suse as then did not
> Support ovs 2.0 or there kernel ovs module did not support patchports.
> As far as I aware  there is no major linux os version that does not have patch
> Support in ovs and also meets the minimum python version of 2.7 required
> by OpenStack So this functionality could safely be removed.
> 

Ok, we should follow up on this to have it deprecated for removal in Ocata.

> >
> > >
> > > >it seems that the LinuxBridge implementation can simply use an L2
> > > >agent extension for creating the vlan interfaces for the subports
> > >
> > > LinuxBridge implementation is the same regardless of the strategy
> > > for OVS. The whole reason we have to come up with these alternative
> > > approaches for OVS is because we can't use the obvious architecture
> > > of letting it plug into the integration bridge due to VLANs already
> > > being used for network isolation. I'm not sure pushing complexity
> > > out to os-vif to deal with this is a great long-term strategy.
> >
> > The complexity we'd be pushing out to os-vif is not much worse than
> > the current complexity of the hybrid_ovs strategy already in place today.
> [Mooney, Sean K] I don’t think strategy 1 is the correct course Of action 
> long-
> term with the trunk bridge approch. I honestly think that The patch port
> creation should be the responsibility of the ovs agent alone.
> 
> I think the DRY principle applies in this respect also. The ovs agent will Be
> required to add or remove patch ports after the vm is booted if subports Are
> added/removed from the truck port. I don’t think it make sense to Write the
> code to do that both in the ovs agent and separately in os-vif.
> 
> Having os-vif simply create the bridge if it does not exist and Add the port 
> to
> it is a much simpler solution in that respect as you can reuse The patch port
> code that is already in neutron and not duplicate it in os-vif.
> https://github.com/openstack/neutron/blob/master/neutron/agent/comm
> on/ovs_lib.py#L368-L371

I don't think we'd be in too much danger in terms of DRY for creating a patch 
port pair in os-vif, because it already has [1] from Nova which is just some 
basic wrapping around a shell command. [1] could be extended to allow creating 
an OVS patch port rather than just a regular port, and I think we can rely on 
the shell command to not produce any bugs that would need fixed in both places.

[1] 
https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/linux_net.py#L49-L60

> 
> 
> >
> > >
> > > >Also, we didn’t make the OVS agent monitor for new linux bridges in
> > > >the hybrid_ovs strategy so that Neutron could be responsible for
> > > >creating the veth pair.
> > >
> > > Linux Bridges are outside of the domain of OVS and even its agent. The
> > > L2 agent doesn't actually do anything with the bridge itself, it just
> > > needs a veth device it can put iptables rules on. That's in contrast
> > > to these new OVS bridges that we will be managing rules for, creating
> > > additional patch ports, etc.
> >
> > I wouldn't say linux bridges are totally outside of its domain because
> > it relies on them for security groups. Rather than relying on an
> > arbitrary naming convention between Neutron and Nova, we could've
> > implemented monitoring 

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Peters, Rawlin]

On Tuesday, June 14, 2016 6:27 PM, Kevin Benton (ke...@benton.pub) wrote:
> >which generates an arbitrary name
>
> I'm not a fan of this approach because it requires coordinated assumptions.
> With the OVS hybrid plug strategy we have to make guesses on the agent side
> about the presence of bridges with specific names that we never explicitly
> requested and that we were never explicitly told about. So we end up with code
> like [1] that is looking for a particular end of a veth pair it just hopes is
> there so the rules have an effect.

I don't think this should be viewed as a downside of Strategy 1 because, at
least when we use patch port pairs, we can easily get the peer name from the
port on br-int, then use the equivalent of "ovs-vsctl iface-to-br "
to get the name of the bridge. If we allow supporting veth pairs to implement
the subports, then getting the arbitrary trunk bridge/veth names isn't as
trivial.

This also brings up the question: do we even need to support veth pairs over
patch port pairs anymore? Are there any distros out there that support
openstack but not OVS patch ports?

>
> >it seems that the LinuxBridge implementation can simply use an L2 agent
> >extension for creating the vlan interfaces for the subports
>
> LinuxBridge implementation is the same regardless of the strategy for OVS. The
> whole reason we have to come up with these alternative approaches for OVS is
> because we can't use the obvious architecture of letting it plug into the
> integration bridge due to VLANs already being used for network isolation. I'm
> not sure pushing complexity out to os-vif to deal with this is a great
> long-term strategy.

The complexity we'd be pushing out to os-vif is not much worse than the current
complexity of the hybrid_ovs strategy already in place today.

>
> >Also, we didn’t make the OVS agent monitor for new linux bridges in the
> >hybrid_ovs strategy so that Neutron could be responsible for creating the 
> >veth
> >pair.
>
> Linux Bridges are outside of the domain of OVS and even its agent. The L2 
> agent
> doesn't actually do anything with the bridge itself, it just needs a veth
> device it can put iptables rules on. That's in contrast to these new OVS
> bridges that we will be managing rules for, creating additional patch ports,
> etc.

I wouldn't say linux bridges are totally outside of its domain because it relies
on them for security groups. Rather than relying on an arbitrary naming
convention between Neutron and Nova, we could've implemented monitoring for new
linux bridges to create veth pairs and firewall rules on. I'm glad we didn't,
because that logic is specific to that particular firewall driver, similar to
how this trunk bridge monitoring would be specific to only vlan-aware-vms. I
think the logic lives best within an L2 agent extension, outside of the core
of the OVS agent.

>
> >Why shouldn't we use the tools that are already available to us?
>
> Because we're trying to build a house and all we have are paint brushes. :)

To me it seems like we already have a house that just needs a little paint :)

>
>
> 1.
> https://github.com/openstack/neutron/blob/f78e5b4ec812cfcf5ab8b50fca62d1ae0dd7741d/neutron/agent/linux/iptables_firewall.py#L919-L923
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Peters, Rawlin]

On Tuesday, June 14, 2016 3:43 AM, Daniel P. Berrange (berra...@redhat.com) 
wrote:
> 
> On Tue, Jun 14, 2016 at 02:35:57AM -0700, Kevin Benton wrote:
> > In strategy 2 we just pass 1 bridge name to Nova. That's the one that
> > is ensures is created and plumbs the VM to. Since it's not responsible
> > for patch ports it doesn't need to know anything about the other bridge.
> 
> Ok, so we're already passing that bridge name - all we need change is make
> sure it is actuall created if it doesn't already exist ? If so that sounds 
> simple
> enough to add to os-vif - we already have exactly the same logic for the
> linux_bridge plugin

Neutron doesn't actually pass the bridge name in the vif_details today, but 
Nova will use that bridge rather than br-int if it's passed in the vif_details.

In terms of strategy 1, I was still only envisioning one bridge name getting 
passed in the vif_details (br-int). The "plug" action is only a variation of 
the hybrid_ovs strategy I mentioned earlier, which generates an arbitrary name 
for the linux bridge, uses that bridge in the instance's libvert XML config 
file, then creates a veth pair between the linux bridge and br-int. Like 
hybrid_ovs, the only bridge Nova/os-vif needs to know about is br-int for 
Strategy 1.

In terms of architecture, we get KISS with Strategy 1 (W.R.T. the OVS agent, 
which is the most complex piece of this IMO). Using an L2 agent extension, we 
will also get DRY as well because it seems that the LinuxBridge implementation 
can simply use an L2 agent extension for creating the vlan interfaces for the 
subports. Similar to how QoS has different drivers for its L2 agent extension, 
we could have different drivers for OVS and LinuxBridge within the 'trunk' L2 
agent extension. Each driver will want to make use of the same RPC calls/push 
mechanisms for subport creation/deletion.

Also, we didn’t make the OVS agent monitor for new linux bridges in the 
hybrid_ovs strategy so that Neutron could be responsible for creating the veth 
pair. Was that a mistake or just an instance of KISS? Why shouldn't we use the 
tools that are already available to us?

Regards,
Rawlin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Peters, Rawlin]

On Monday, June 13, 2016 6:28 AM, Daniel P. Berrange wrote:
> 
> On Mon, Jun 13, 2016 at 07:39:29AM -0400, Assaf Muller wrote:
> > On Mon, Jun 13, 2016 at 4:35 AM, Daniel P. Berrange
>  wrote:
> > > On Thu, Jun 09, 2016 at 05:31:13PM -0600, Carl Baldwin wrote:
> > >> Hi,
> > >>
> > >> You may or may not be aware of the vlan-aware-vms effort [1] in
> > >> Neutron.  If not, there is a spec and a fair number of patches in
> > >> progress for this.  Essentially, the goal is to allow a VM to
> > >> connect to multiple Neutron networks by tagging traffic on a single
> > >> port with VLAN tags.
> > >>
> > >> This effort will have some effect on vif plugging because the
> > >> datapath will include some changes that will effect how vif
> > >> plugging is done today.
> > >>
> > >> The design proposal for trunk ports with OVS adds a new bridge for
> > >> each trunk port.  This bridge will demux the traffic and then
> > >> connect to br-int with patch ports for each of the networks.
> > >> Rawlin Peters has some ideas for expanding the vif capability to
> > >> include this wiring.
> > >>
> > >> There is also a proposal for connecting to linux bridges by using
> > >> kernel vlan interfaces.
> > >>
> > >> This effort is pretty important to Neutron in the Newton timeframe.
> > >> I wanted to send this out to start rounding up the reviewers and
> > >> other participants we need to see how we can start putting together
> > >> a plan for nova integration of this feature (via os-vif?).
> > >
> > > I've not taken a look at the proposal, but on the timing side of
> > > things it is really way to late to start this email thread asking
> > > for design input from os-vif or nova. We're way past the spec
> > > proposal deadline for Nova in the Newton cycle, so nothing is going
> > > to happen until the Ocata cycle no matter what Neutron want  in
> > > Newton. For os-vif our focus right now is exclusively on getting
> > > existing functionality ported over, and integrated into Nova in
> > > Newton. So again we're not really looking to spend time on further os-vif
> design work right now.
> > >
> > > In the Ocata cycle we'll be looking to integrate os-vif into Neutron
> > > to let it directly serialize VIF objects and send them over to Nova,
> > > instead of using the ad-hoc port-binding dicts.  From the Nova side,
> > > we're not likely to want to support any new functionality that
> > > affects port-binding data until after Neutron is converted to
> > > os-vif. So Ocata at the earliest, but probably more like P,
> > > unless the Neutron conversion to os-vif gets completed unexpectedly
> quickly.
> >
> > In light of this feature being requested by the NFV, container and
> > baremetal communities, and that Neutron's os-vif integration work
> > hasn't begun, does it make sense to block Nova VIF work? Are we
> > comfortable, from a wider OpenStack perspective, to wait until
> > possibly the P release? I think it's our collective responsibility as
> > developers to find creative ways to meet deadlines, not serializing
> > work on features and letting processes block us.
> 
> Everyone has their own personal set of features that are their personal
> priority items. Nova evaluates all the competing demands and decides on
> what the project's priorities are for the given cycle. For Newton Nova's
> priority is to convert existing VIF functionality to use os-vif. Anything 
> else vif
> related takes a backseat to this project priority. This formal modelling of 
> VIFs
> and developing a plugin facility has already been strung out over at least 3
> release cycles now. We're finally in a position to get it completed, and we're
> not going to divert attention away from this, to other new features requests
> until its done as that'll increase the chances of it getting strung out for 
> yet
> another release which is in no ones interests.

I think we are all in agreement that integrating os-vif into Nova during the 
Newton cycle is the highest priority.

The question is, once os-vif has been integrated into Nova are we going to have 
any problem augmenting the current os-vif OvsPlugin in order to support 
vlan-aware-vms in the Newton release? Based upon the current Nova integration 
patch [1] I believe that any vif-plugging changes required to implement 
vlan-aware-vms could be entirely localized to the os-vif OvsPlugin, so Nova 
wouldn't directly need to be involved there.

That said, there are currently a couple of vif-plugging strategies we could go 
with for wiring trunk ports for OVS, each of them requiring varying levels of 
os-vif augmentation:
Strategy 1) When Nova is plugging a trunk port, it creates the OVS trunk 
bridge, attaches the tap to it, and creates one patch port pair from the trunk 
bridge to br-int.
Strategy 2) Neutron passes a bridge name to Nova, and Nova uses this bridge 
name to create the OVS trunk bridge and attach the tap to it (no patch port 
pair plugging into br-int).

Strategy 1 requires 

Re: [openstack-dev] [neutron][ovs] The way we deal with MTU

[Peters, Rawlin]

Hi Ihar,

This reminds me of a mailing list thread from a while back about moving OVS 
ports between namespaces being considered harmful [1]. Do you know if that was 
ever resolved by the OVS folks? Or, is this MTU bug just further indication of 
this action being harmful?

Another comment inline.

Rawlin Peters

[1] http://lists.openstack.org/pipermail/openstack-dev/2015-February/056834.html

On  Monday, June 13, 2016 10:50 AM, Ihar Hrachyshka wrote:
> 
> Hi all,
> 
> in Mitaka, we introduced a bunch of changes to the way we handle MTU in
> Neutron/Nova, making sure that the whole instance data path, starting from
> instance internal interface, thru hybrid bridge, into the br-int; as well as
> router data path (qr) have proper MTU value set on all participating devices.
> On hypervisor side, both Nova and Neutron take part in it, setting it with ip-
> link tool based on what Neutron plugin calculates for us. So far so good.
> 
> Turns out that for OVS, it does not work as expected in regards to br-int.
> There was a bug reported lately: https://launchpad.net/bugs/1590397
> 
> Briefly, when we try to set MTU on a device that is plugged into a bridge, and
> if the bridge already has another port with lower MTU, the bridge itself
> inherits MTU from that latter port, and Linux kernel (?) does not allow to set
> MTU on the first device at all, making ip link calls ineffective.
> 
> AFAIU this behaviour is consistent with Linux bridging rules: you can’t have
> ports of different MTU plugged into the same bridge.
> 
> Now, that’s a huge problem for Neutron, because we plug ports that belong
> to different networks (and that hence may have different MTUs) into the
> same br-int bridge.
> 
> So I played with the code locally a bit and spotted that currently, we set MTU
> for router ports before we move their devices into router namespaces. And
> once the device is in a namespace, ip-link actually works. So I wrote a fix 
> with
> a functional test that proves the point:
> https://review.openstack.org/#/c/327651/ The fix was validated by the
> reporter of the original bug and seems to fix the issue for him.
> 
> It’s suspicious that it works from inside a namespace but not when the
> device is still in the root namespace. So I reached out to Jiri Benc from our
> local Open vSwitch team, and here is a quote:
> 
> ===
> 
> "It's a bug in ovs-vswitchd. It doesn't see the interface that's in other 
> netns
> and thus cannot enforce the correct MTU.
> 
> We'll hopefully fix it and disallow incorrect MTU setting even across
> namespaces. However, it requires significant effort and rework of ovs name
> space handling.
> 
> You should not depend on the current buggy behavior. Don't set MTU of the
> internal interfaces higher than the rest of the bridge, it's not supported.
> Hacking this around by moving the interface to a netns is exploiting of a bug.
> 
> We can certainly discuss whether this limitation could be relaxed.
> Honestly, I don't know, it's for a discussion upstream. But as of now, it's 
> not
> supported and you should not do it.”
> 
> So basically, as long as we try to plug ports with different MTUs into the 
> same
> bridge, we are utilizing a bug in Open vSwitch, that may break us any time.
> 
> I guess our alternatives are:
> - either redesign bridge setup for openvswitch to e.g. maintain a bridge per
> network;
> - or talk to ovs folks on whether they may support that for us.
> 

It seems like another alternative would be to always use veth devices by 
default rather than internal OVS ports (i.e. ovs_use_veth = True), but that 
would likely mean taking a large performance hit that no one will be happy 
about.

> I understand the former option is too scary. It opens lots of questions,
> including upgrade impact since it will obviously introduce a dataplane
> downtime. That would be a huge shift in paradigm, probably too huge to
> swallow. The latter option may not fly with vswitch folks. Any better ideas?
> 
> It’s also not clear whether we want to proceed with my immediate fix.
> Advices are welcome.
> 
> Thanks,
> Ihar
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev