Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
I played adding some flows for forwarding as vlan packets and it worked(i.e compute and gateway communicated through vlan packets on both sides) I created a vm on vlan network 172.16.1.1/24 with vlan id 20 and an external network 10.0.0.0/24 to reach internet. In the current code, when I tired to reach external network(8.8.8.8) from this vm, below physical flows will be run for routing and redirecting the packet through geneve tunnel. table=15 ip,metadata=0x2 actions=dec_ttl(),load:0xa01->NXM_NX_XXREG0[96..127],load:0xa65->NXM_NX_XXREG0[64..95],set_field:fa:16:3e:49:bb:60->eth_src,set_field:0x3->reg15,load:0x1->NXM_NX_REG10[0],resubmit(,16) //routing table=17 reg15=0x3,metadata=0x2 actions=set_field:0x4->reg15,resubmit(,18) //redirect table=32, priority=100,reg15=0x4,metadata=0x2 actions=load:0x2->NXM_NX_TUN_ID[0..23],set_field:0x4->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],bundle(eth_src,0,active_backup,ofport,slaves:3,4) When I manually added below flows (in same tables) with higher priority for vlans, packet is forwarded with vlan tag instead of tunneling. sudo ovs-ofctl add-flow br-int "table=15,ip,priority=49,metadata=0x2,dl_dst=fa:16:3e:8c:02:5d,actions=dec_ttl(),load:0xac100201->NXM_NX_REG0[],load:0x3->NXM_NX_REG15[],load:0x1->NXM_NX_REG10[0],resubmit(,16)" sudo ovs-ofctl add-flow br-int "table=16,priority=100,reg0=0xac100201,reg15=0x3,metadata=0x2,action=mod_dl_dst:fa:16:3e:8c:02:5d,resubmit(,17)" sudo ovs-ofctl add-flow br-int "table=32,priority=200,reg0=0xac100201,reg15=0x4,metadata=0x2,action=mod_vlan_vid:20,output:9,strip_vlan" Note: Above table 15 flow is added by ovn-controller for each vlan network(fa:16:3e:8c:02:5d is 172.16.1.1 MAC) I will work on ovn-controller changes to have these flows. Thanks Anil On Tue, Apr 17, 2018 at 2:20 AM, Ben Pfaff wrote: > On Fri, Apr 13, 2018 at 09:22:44PM -0400, Russell Bryant wrote: > > On Fri, Apr 13, 2018 at 9:01 PM, Russell Bryant wrote: > > > On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: > > >> On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: > > >>> vm created on a vlan tenant network is using geneve tunnel(between > compute > > >>> and gateway nodes) to reach external network. Because of this, we > need to > > >>> consider tunnelling overhead while assigning MTU for vlan network. > Can we > > >>> improve OVN to avoid tunnelling in this case. > > >> > > >> When OVN tunnels packets, the tunnel metadata includes information on > > >> the logical network, logical input port, and logical output port. The > > >> logical input port is only used for egress ACLs, so it could be > omitted > > >> if egress ACLs are constrained not to match on the logical input port. > > >> The logical network and logical output port are still needed, though, > so > > >> to encode that in a VLAN they would have to add up to 12 bits or less. > > >> That's pretty constraining. Do you have some idea for how to do it? > > > > > > I don't think ACLs are a factor here because it's actually the logical > > > router pipeline forwarded the packet over a tunnel. The only logical > > > switches involved are VLAN networks (a switch with a localnet port). > > > > > > The unexpected behavior here is that despite using all VLAN networks, > > > a Geneve tunnel is used when the packet is sent to the L3 gateway node > > > that's doing SNAT. Note that the type of router configured here is > > > the hybrid-type, where routing is fully distributed in all cases > > > except when NAT is required, then it gets redirected to a central > > > point. That redirect is what we're seeing here. > > > > > > I've thought of two ways out of this: > > > > > > 1) In this scenario, if you really don't want any tunneling in use, > > > configure a fully centralized router instead. The downside is that > > > East-West routing will be centralized, as well. > > > > > > 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed > > > east-west router <-> ls2 (VLAN) used just to interconnect the routers > > > <-> centralized router for SNAT. The downside here is that you still > > > lose the ability to bind floating IPs directly to compute nodes like > > > today. > > > > > > 3) Figure out a way for OVN to do this redirect to the gateway host > > > over a VLAN network. I suspect this isn't trivial and honestly > > > haven't spent the time to figure out what it would take, but this does > > > seem like the ideal behavior. > > > > > > -- > > > Russell Bryant > > > > After an internal conversation on this topic, I wrote the following > > doc to summarize what was observed and to capture ideas for next > > steps: > > > > https://docs.google.com/document/d/1JecGIXPH0RAqfGvD0nmtBdEU1zflH > ACp8WSRnKCFSgg/edit?usp=sharing > > I wonder whether this should be added to the OVN FAQ I starting writing > earlier today: > https://patchwork.ozlabs.org/patch/898868/ > ___ discuss mailing list
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Fri, Apr 13, 2018 at 09:22:44PM -0400, Russell Bryant wrote: > On Fri, Apr 13, 2018 at 9:01 PM, Russell Bryant wrote: > > On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: > >> On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: > >>> vm created on a vlan tenant network is using geneve tunnel(between compute > >>> and gateway nodes) to reach external network. Because of this, we need to > >>> consider tunnelling overhead while assigning MTU for vlan network. Can we > >>> improve OVN to avoid tunnelling in this case. > >> > >> When OVN tunnels packets, the tunnel metadata includes information on > >> the logical network, logical input port, and logical output port. The > >> logical input port is only used for egress ACLs, so it could be omitted > >> if egress ACLs are constrained not to match on the logical input port. > >> The logical network and logical output port are still needed, though, so > >> to encode that in a VLAN they would have to add up to 12 bits or less. > >> That's pretty constraining. Do you have some idea for how to do it? > > > > I don't think ACLs are a factor here because it's actually the logical > > router pipeline forwarded the packet over a tunnel. The only logical > > switches involved are VLAN networks (a switch with a localnet port). > > > > The unexpected behavior here is that despite using all VLAN networks, > > a Geneve tunnel is used when the packet is sent to the L3 gateway node > > that's doing SNAT. Note that the type of router configured here is > > the hybrid-type, where routing is fully distributed in all cases > > except when NAT is required, then it gets redirected to a central > > point. That redirect is what we're seeing here. > > > > I've thought of two ways out of this: > > > > 1) In this scenario, if you really don't want any tunneling in use, > > configure a fully centralized router instead. The downside is that > > East-West routing will be centralized, as well. > > > > 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed > > east-west router <-> ls2 (VLAN) used just to interconnect the routers > > <-> centralized router for SNAT. The downside here is that you still > > lose the ability to bind floating IPs directly to compute nodes like > > today. > > > > 3) Figure out a way for OVN to do this redirect to the gateway host > > over a VLAN network. I suspect this isn't trivial and honestly > > haven't spent the time to figure out what it would take, but this does > > seem like the ideal behavior. > > > > -- > > Russell Bryant > > After an internal conversation on this topic, I wrote the following > doc to summarize what was observed and to capture ideas for next > steps: > > https://docs.google.com/document/d/1JecGIXPH0RAqfGvD0nmtBdEU1zflHACp8WSRnKCFSgg/edit?usp=sharing I wonder whether this should be added to the OVN FAQ I starting writing earlier today: https://patchwork.ozlabs.org/patch/898868/ ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Fri, Apr 13, 2018 at 6:22 PM, Russell Bryant wrote: > > On Fri, Apr 13, 2018 at 9:01 PM, Russell Bryant wrote: > > On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: > >> On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: > >>> vm created on a vlan tenant network is using geneve tunnel(between compute > >>> and gateway nodes) to reach external network. Because of this, we need to > >>> consider tunnelling overhead while assigning MTU for vlan network. Can we > >>> improve OVN to avoid tunnelling in this case. > >> > >> When OVN tunnels packets, the tunnel metadata includes information on > >> the logical network, logical input port, and logical output port. The > >> logical input port is only used for egress ACLs, so it could be omitted > >> if egress ACLs are constrained not to match on the logical input port. > >> The logical network and logical output port are still needed, though, so > >> to encode that in a VLAN they would have to add up to 12 bits or less. > >> That's pretty constraining. Do you have some idea for how to do it? > > > > I don't think ACLs are a factor here because it's actually the logical > > router pipeline forwarded the packet over a tunnel. The only logical > > switches involved are VLAN networks (a switch with a localnet port). > > > > The unexpected behavior here is that despite using all VLAN networks, > > a Geneve tunnel is used when the packet is sent to the L3 gateway node > > that's doing SNAT. Note that the type of router configured here is > > the hybrid-type, where routing is fully distributed in all cases > > except when NAT is required, then it gets redirected to a central > > point. That redirect is what we're seeing here. > > > > I've thought of two ways out of this: > > > > 1) In this scenario, if you really don't want any tunneling in use, > > configure a fully centralized router instead. The downside is that > > East-West routing will be centralized, as well. > > > > 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed > > east-west router <-> ls2 (VLAN) used just to interconnect the routers > > <-> centralized router for SNAT. The downside here is that you still > > lose the ability to bind floating IPs directly to compute nodes like > > today. > > > > 3) Figure out a way for OVN to do this redirect to the gateway host > > over a VLAN network. I suspect this isn't trivial and honestly > > haven't spent the time to figure out what it would take, but this does > > seem like the ideal behavior. For the VLAN network to be used to redirect to a gateway, direct physical L2 connection is required, but Gateway router in OVN generally doesn't require L2 connection. > > > > -- > > Russell Bryant > > After an internal conversation on this topic, I wrote the following > doc to summarize what was observed and to capture ideas for next > steps: > > https://docs.google.com/document/d/1JecGIXPH0RAqfGvD0nmtBdEU1zflHACp8WSRnKCFSgg/edit?usp=sharing > > -- > Russell Bryant > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Fri, Apr 13, 2018 at 9:01 PM, Russell Bryant wrote: > On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: >> On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: >>> vm created on a vlan tenant network is using geneve tunnel(between compute >>> and gateway nodes) to reach external network. Because of this, we need to >>> consider tunnelling overhead while assigning MTU for vlan network. Can we >>> improve OVN to avoid tunnelling in this case. >> >> When OVN tunnels packets, the tunnel metadata includes information on >> the logical network, logical input port, and logical output port. The >> logical input port is only used for egress ACLs, so it could be omitted >> if egress ACLs are constrained not to match on the logical input port. >> The logical network and logical output port are still needed, though, so >> to encode that in a VLAN they would have to add up to 12 bits or less. >> That's pretty constraining. Do you have some idea for how to do it? > > I don't think ACLs are a factor here because it's actually the logical > router pipeline forwarded the packet over a tunnel. The only logical > switches involved are VLAN networks (a switch with a localnet port). > > The unexpected behavior here is that despite using all VLAN networks, > a Geneve tunnel is used when the packet is sent to the L3 gateway node > that's doing SNAT. Note that the type of router configured here is > the hybrid-type, where routing is fully distributed in all cases > except when NAT is required, then it gets redirected to a central > point. That redirect is what we're seeing here. > > I've thought of two ways out of this: > > 1) In this scenario, if you really don't want any tunneling in use, > configure a fully centralized router instead. The downside is that > East-West routing will be centralized, as well. > > 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed > east-west router <-> ls2 (VLAN) used just to interconnect the routers > <-> centralized router for SNAT. The downside here is that you still > lose the ability to bind floating IPs directly to compute nodes like > today. > > 3) Figure out a way for OVN to do this redirect to the gateway host > over a VLAN network. I suspect this isn't trivial and honestly > haven't spent the time to figure out what it would take, but this does > seem like the ideal behavior. > > -- > Russell Bryant After an internal conversation on this topic, I wrote the following doc to summarize what was observed and to capture ideas for next steps: https://docs.google.com/document/d/1JecGIXPH0RAqfGvD0nmtBdEU1zflHACp8WSRnKCFSgg/edit?usp=sharing -- Russell Bryant ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: > On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: >> vm created on a vlan tenant network is using geneve tunnel(between compute >> and gateway nodes) to reach external network. Because of this, we need to >> consider tunnelling overhead while assigning MTU for vlan network. Can we >> improve OVN to avoid tunnelling in this case. > > When OVN tunnels packets, the tunnel metadata includes information on > the logical network, logical input port, and logical output port. The > logical input port is only used for egress ACLs, so it could be omitted > if egress ACLs are constrained not to match on the logical input port. > The logical network and logical output port are still needed, though, so > to encode that in a VLAN they would have to add up to 12 bits or less. > That's pretty constraining. Do you have some idea for how to do it? I don't think ACLs are a factor here because it's actually the logical router pipeline forwarded the packet over a tunnel. The only logical switches involved are VLAN networks (a switch with a localnet port). The unexpected behavior here is that despite using all VLAN networks, a Geneve tunnel is used when the packet is sent to the L3 gateway node that's doing SNAT. Note that the type of router configured here is the hybrid-type, where routing is fully distributed in all cases except when NAT is required, then it gets redirected to a central point. That redirect is what we're seeing here. I've thought of two ways out of this: 1) In this scenario, if you really don't want any tunneling in use, configure a fully centralized router instead. The downside is that East-West routing will be centralized, as well. 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed east-west router <-> ls2 (VLAN) used just to interconnect the routers <-> centralized router for SNAT. The downside here is that you still lose the ability to bind floating IPs directly to compute nodes like today. 3) Figure out a way for OVN to do this redirect to the gateway host over a VLAN network. I suspect this isn't trivial and honestly haven't spent the time to figure out what it would take, but this does seem like the ideal behavior. -- Russell Bryant ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: > vm created on a vlan tenant network is using geneve tunnel(between compute > and gateway nodes) to reach external network. Because of this, we need to > consider tunnelling overhead while assigning MTU for vlan network. Can we > improve OVN to avoid tunnelling in this case. When OVN tunnels packets, the tunnel metadata includes information on the logical network, logical input port, and logical output port. The logical input port is only used for egress ACLs, so it could be omitted if egress ACLs are constrained not to match on the logical input port. The logical network and logical output port are still needed, though, so to encode that in a VLAN they would have to add up to 12 bits or less. That's pretty constraining. Do you have some idea for how to do it? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss