Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Fri, Apr 13, 2018 at 6:22 PM, Russell Bryant wrote: > > On Fri, Apr 13, 2018 at 9:01 PM, Russell Bryant wrote: > > On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: > >> On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: > >>> vm created on a vlan tenant network is using geneve tunnel(between compute > >>> and gateway nodes) to reach external network. Because of this, we need to > >>> consider tunnelling overhead while assigning MTU for vlan network. Can we > >>> improve OVN to avoid tunnelling in this case. > >> > >> When OVN tunnels packets, the tunnel metadata includes information on > >> the logical network, logical input port, and logical output port. The > >> logical input port is only used for egress ACLs, so it could be omitted > >> if egress ACLs are constrained not to match on the logical input port. > >> The logical network and logical output port are still needed, though, so > >> to encode that in a VLAN they would have to add up to 12 bits or less. > >> That's pretty constraining. Do you have some idea for how to do it? > > > > I don't think ACLs are a factor here because it's actually the logical > > router pipeline forwarded the packet over a tunnel. The only logical > > switches involved are VLAN networks (a switch with a localnet port). > > > > The unexpected behavior here is that despite using all VLAN networks, > > a Geneve tunnel is used when the packet is sent to the L3 gateway node > > that's doing SNAT. Note that the type of router configured here is > > the hybrid-type, where routing is fully distributed in all cases > > except when NAT is required, then it gets redirected to a central > > point. That redirect is what we're seeing here. > > > > I've thought of two ways out of this: > > > > 1) In this scenario, if you really don't want any tunneling in use, > > configure a fully centralized router instead. The downside is that > > East-West routing will be centralized, as well. > > > > 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed > > east-west router <-> ls2 (VLAN) used just to interconnect the routers > > <-> centralized router for SNAT. The downside here is that you still > > lose the ability to bind floating IPs directly to compute nodes like > > today. > > > > 3) Figure out a way for OVN to do this redirect to the gateway host > > over a VLAN network. I suspect this isn't trivial and honestly > > haven't spent the time to figure out what it would take, but this does > > seem like the ideal behavior. For the VLAN network to be used to redirect to a gateway, direct physical L2 connection is required, but Gateway router in OVN generally doesn't require L2 connection. > > > > -- > > Russell Bryant > > After an internal conversation on this topic, I wrote the following > doc to summarize what was observed and to capture ideas for next > steps: > > https://docs.google.com/document/d/1JecGIXPH0RAqfGvD0nmtBdEU1zflHACp8WSRnKCFSgg/edit?usp=sharing > > -- > Russell Bryant > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Fri, Apr 13, 2018 at 9:01 PM, Russell Bryant wrote: > On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: >> On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: >>> vm created on a vlan tenant network is using geneve tunnel(between compute >>> and gateway nodes) to reach external network. Because of this, we need to >>> consider tunnelling overhead while assigning MTU for vlan network. Can we >>> improve OVN to avoid tunnelling in this case. >> >> When OVN tunnels packets, the tunnel metadata includes information on >> the logical network, logical input port, and logical output port. The >> logical input port is only used for egress ACLs, so it could be omitted >> if egress ACLs are constrained not to match on the logical input port. >> The logical network and logical output port are still needed, though, so >> to encode that in a VLAN they would have to add up to 12 bits or less. >> That's pretty constraining. Do you have some idea for how to do it? > > I don't think ACLs are a factor here because it's actually the logical > router pipeline forwarded the packet over a tunnel. The only logical > switches involved are VLAN networks (a switch with a localnet port). > > The unexpected behavior here is that despite using all VLAN networks, > a Geneve tunnel is used when the packet is sent to the L3 gateway node > that's doing SNAT. Note that the type of router configured here is > the hybrid-type, where routing is fully distributed in all cases > except when NAT is required, then it gets redirected to a central > point. That redirect is what we're seeing here. > > I've thought of two ways out of this: > > 1) In this scenario, if you really don't want any tunneling in use, > configure a fully centralized router instead. The downside is that > East-West routing will be centralized, as well. > > 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed > east-west router <-> ls2 (VLAN) used just to interconnect the routers > <-> centralized router for SNAT. The downside here is that you still > lose the ability to bind floating IPs directly to compute nodes like > today. > > 3) Figure out a way for OVN to do this redirect to the gateway host > over a VLAN network. I suspect this isn't trivial and honestly > haven't spent the time to figure out what it would take, but this does > seem like the ideal behavior. > > -- > Russell Bryant After an internal conversation on this topic, I wrote the following doc to summarize what was observed and to capture ideas for next steps: https://docs.google.com/document/d/1JecGIXPH0RAqfGvD0nmtBdEU1zflHACp8WSRnKCFSgg/edit?usp=sharing -- Russell Bryant ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Fri, Apr 13, 2018 at 5:27 PM, Ben Pfaff wrote: > On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: >> vm created on a vlan tenant network is using geneve tunnel(between compute >> and gateway nodes) to reach external network. Because of this, we need to >> consider tunnelling overhead while assigning MTU for vlan network. Can we >> improve OVN to avoid tunnelling in this case. > > When OVN tunnels packets, the tunnel metadata includes information on > the logical network, logical input port, and logical output port. The > logical input port is only used for egress ACLs, so it could be omitted > if egress ACLs are constrained not to match on the logical input port. > The logical network and logical output port are still needed, though, so > to encode that in a VLAN they would have to add up to 12 bits or less. > That's pretty constraining. Do you have some idea for how to do it? I don't think ACLs are a factor here because it's actually the logical router pipeline forwarded the packet over a tunnel. The only logical switches involved are VLAN networks (a switch with a localnet port). The unexpected behavior here is that despite using all VLAN networks, a Geneve tunnel is used when the packet is sent to the L3 gateway node that's doing SNAT. Note that the type of router configured here is the hybrid-type, where routing is fully distributed in all cases except when NAT is required, then it gets redirected to a central point. That redirect is what we're seeing here. I've thought of two ways out of this: 1) In this scenario, if you really don't want any tunneling in use, configure a fully centralized router instead. The downside is that East-West routing will be centralized, as well. 2) Use two routers. ls1 (VLAN) with all ports for VMs <-> distributed east-west router <-> ls2 (VLAN) used just to interconnect the routers <-> centralized router for SNAT. The downside here is that you still lose the ability to bind floating IPs directly to compute nodes like today. 3) Figure out a way for OVN to do this redirect to the gateway host over a VLAN network. I suspect this isn't trivial and honestly haven't spent the time to figure out what it would take, but this does seem like the ideal behavior. -- Russell Bryant ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN vm on vlan network using geneve tunnel for external traffic
On Wed, Apr 11, 2018 at 07:44:25PM +0530, Anil Venkata wrote: > vm created on a vlan tenant network is using geneve tunnel(between compute > and gateway nodes) to reach external network. Because of this, we need to > consider tunnelling overhead while assigning MTU for vlan network. Can we > improve OVN to avoid tunnelling in this case. When OVN tunnels packets, the tunnel metadata includes information on the logical network, logical input port, and logical output port. The logical input port is only used for egress ACLs, so it could be omitted if egress ACLs are constrained not to match on the logical input port. The logical network and logical output port are still needed, though, so to encode that in a VLAN they would have to add up to 12 bits or less. That's pretty constraining. Do you have some idea for how to do it? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] OVN meetup at VMware, May 15
In the OVN meeting this week, we discussed doing an in-person meetup of OVN developers and other interested people on May 15. I volunteered to host at VMware. I suggest 2pm to 4pm. If you're interested in attending, please reply to me or to the thread. If people want to attend remotely, we can conference them in, so please also let me know if you'd like to attend that way. We also need to schedule speakers or discussion, If you'd like to speak about something, or hear about something, or propose a topic, please speak up. Thanks, Ben. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] There's no available (non-isolated) pmd thread on numa node 0, Expect reduced performance.
Hi Ian, > As you are setting all lcore and pmd core to node 1 why are you giving 1024 memory to node 0? > When processing packets for this port it means cpu is accessing data across the numa nodes which causes a performance penalty I am benchmarking performance in different settings and trying to understand the roles of OVS and DPDK in mediating core affinity of pmds and hugepage utilization. Your answer helps a lot! > try using ‘other_config:dpdk-socket-mem=0,4096’ and see if you still see the issue. But this warning should appear regardless the socket-mem allocation right? If my understanding is correct, when OVS pmd's are pinned to 10-19 and the VM tespmd app is pinned to core 2; the OVSpmd thread running on node 1 is having to access a huge-page on node 0 which VMtestpmd happens to access as well. Thanks, Alan 2018-04-12 18:28 GMT+02:00 Stokes, Ian : > Hi, > > > > I was able to reproduce the issue on my system. > > > > As you are setting all lcore and pmd core to node 1 why are you giving > 1024 memory to node 0? > > > > I saw the same issue on my system but the warning did not appear once > memory was allocated to node 1 only. > > > > I would think the VM being launched is using memory for the vhost port > from node 0, however the queue for the vhost port is assigned to core 14 > which is on node1. When processing packets for this port it means cpu is > accessing data across the numa nodes which causes a performance penalty, > hence the warning. > > > > To avoid you should ensure all memory and cores operate on the same node > where possible, try using ‘other_config:dpdk-socket-mem=0,4096’ and see > if you still see the issue. > > > > Thanks > > Ian > > > > *From:* ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss-bounces@ > openvswitch.org] *On Behalf Of *Alan Kayahan > *Sent:* Thursday, April 12, 2018 2:27 AM > *To:* ovs-discuss@openvswitch.org > *Subject:* [ovs-discuss] There's no available (non-isolated) pmd thread > on numa node 0, Expect reduced performance. > > > > Hello, > > > > On the following setup, where all cores but 0 are isolated, > > > > available: 2 nodes (0-1) > > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 > > node 1 cpus: 10 11 12 13 14 15 16 17 18 19 > > > > I am trying to start OVS entirely on numa node 1 as following > > > > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true > other_config:dpdk-lcore-mask=0x00400 other_config:pmd-cpu-mask=0xffc00 > other_config:dpdk-socket-mem=1024,4096 > > > > However when I create a vhost port SRC to attach a VNF(via virtio) on node > 0, I get the following > > > > dpif_netdev|WARN|There's no available (non-isolated) pmd thread on numa > node 0. Queue 0 on port 'SRC' will be assigned to the pmd on core 14 (numa > node 1). Expect reduced performance. > > > > Any ideas? > > > > Thanks > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
