Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
I think I might have found out the root cause for this delay problem (emphasis on might have). There are two types of masks assigned to OVS, pmd-mask to run PMD threads and ovs-lcore-mak which is used by OVS main thread, handlers and revalidators. The documentation says that number of handler should be total CPUs minus revalidators, so it picks up 13 revalidators and 35 handlers for my system (48 vCPUs), but when I use "tuna -t ovs-vswitchd -CP" they are all just using one vCPU (first one in the ovs-lcore-mask). Shouldn't the number of revalidators and handlers depend on the number of lcores? And they should all be scheduled on different CPUs? I have not tested changing the number of handlers and revalidators and observing that effect yet. Please let me know if I am on right track. -Original Message- From: Muhammad Alp Arslan (alp.ars...@xflowresearch.com) [mailto:alp.ars...@xflowresearch.com] Sent: Thursday, May 10, 2018 11:36 AM To: 'Ben Pfaff' <b...@ovn.org> Cc: 'disc...@openvswitch.org' <disc...@openvswitch.org> Subject: RE: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 So far from what I have observed this is not related to how much traffic is flowing, rather for how much traffic it gets a megaflow miss, and have to be processed by the OF Classifier. If I add a high priority rule that allow/deny all traffic, then I can add as many rules I want with 10-15 ms delay. However, in the real scenario where the traffic pattern is abrupt, and I have to do layer 3 & layer 4 matches to allow/drop certain types of traffic I start seeing a latency that grows exponentially. All I can guess from my limited knowledge is that there is some kind of a lock on the OF tables while the packet is being processed and matched. Would be really helpful if there is a work around for this. I can provide more information, like number of flows, number of flows per table, EMC hits, megaflow hits/miss etc. -Original Message- From: Ben Pfaff [mailto:b...@ovn.org] Sent: Thursday, May 10, 2018 2:31 AM To: alp.ars...@xflowresearch.com Cc: disc...@openvswitch.org Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 Using a controller might yield better latency for adding and removing flows, but I'm a little surprised to hear that there's terribly slow latency when traffic is flowing. Usually, the main thread in OVS (which is the one that does OpenFlow table management) doesn't have much to do, even if there is a high traffic load. I don't have a general-purpose controller to recommend. There are several out there and I imagine that any one of them could be suitable for this project. On Mon, May 07, 2018 at 09:41:17PM +0500, alp.ars...@xflowresearch.com wrote: > What would you recommend as a controller, also would adding rules > using a controller be faster than "ovs-ofctl"? My application > continuously add and delete flows based on the traffic patterns, and > it works fine if there is no traffic flowing through the OVS, but as > soon as I turn on the 40 G links, the time to apply the rules starting > hitting several seconds sometimes up to 100s. It keeps increasing as the existing number of flows increase. > > Can an SDN controller solve this issue? Or is it an inherent OVS > limitation where it takes more time to add rules if more packets are > going to the OF classifier? > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Monday, May 7, 2018 9:36 PM > To: alp.ars...@xflowresearch.com > Cc: disc...@openvswitch.org > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > OVS doesn't have that built in. Usually we think of it as the > responsibility of the controller. > > On Mon, May 07, 2018 at 09:22:47PM +0500, alp.ars...@xflowresearch.com > wrote: > > I mean OpenFlow flows that are persistent across OVS or system restarts. > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Monday, May 7, 2018 9:18 PM > > To: alp.ars...@xflowresearch.com > > Cc: disc...@openvswitch.org > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > 2.9 > > > > What do you mean by "persist the OVS flows"? I have a couple of > > guesses but I'd like to hear from you. > > > > On Mon, May 07, 2018 at 02:17:18AM +0500, > > alp.ars...@xflowresearch.com > > wrote: > > > Thank you Ben for the correction. I am running tests with > > > different scenarios to better understand what's happening inside the OVS-DPDK. > > > One thing that I would like to ask is, is there a way to persist > > > the OVS > > flows? > > > Can OVN help me do that? I don't have any virtual networks, just > > > in and out ports. > >
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
So far from what I have observed this is not related to how much traffic is flowing, rather for how much traffic it gets a megaflow miss, and have to be processed by the OF Classifier. If I add a high priority rule that allow/deny all traffic, then I can add as many rules I want with 10-15 ms delay. However, in the real scenario where the traffic pattern is abrupt, and I have to do layer 3 & layer 4 matches to allow/drop certain types of traffic I start seeing a latency that grows exponentially. All I can guess from my limited knowledge is that there is some kind of a lock on the OF tables while the packet is being processed and matched. Would be really helpful if there is a work around for this. I can provide more information, like number of flows, number of flows per table, EMC hits, megaflow hits/miss etc. -Original Message- From: Ben Pfaff [mailto:b...@ovn.org] Sent: Thursday, May 10, 2018 2:31 AM To: alp.ars...@xflowresearch.com Cc: disc...@openvswitch.org Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 Using a controller might yield better latency for adding and removing flows, but I'm a little surprised to hear that there's terribly slow latency when traffic is flowing. Usually, the main thread in OVS (which is the one that does OpenFlow table management) doesn't have much to do, even if there is a high traffic load. I don't have a general-purpose controller to recommend. There are several out there and I imagine that any one of them could be suitable for this project. On Mon, May 07, 2018 at 09:41:17PM +0500, alp.ars...@xflowresearch.com wrote: > What would you recommend as a controller, also would adding rules > using a controller be faster than "ovs-ofctl"? My application > continuously add and delete flows based on the traffic patterns, and > it works fine if there is no traffic flowing through the OVS, but as > soon as I turn on the 40 G links, the time to apply the rules starting > hitting several seconds sometimes up to 100s. It keeps increasing as the existing number of flows increase. > > Can an SDN controller solve this issue? Or is it an inherent OVS > limitation where it takes more time to add rules if more packets are > going to the OF classifier? > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Monday, May 7, 2018 9:36 PM > To: alp.ars...@xflowresearch.com > Cc: disc...@openvswitch.org > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > OVS doesn't have that built in. Usually we think of it as the > responsibility of the controller. > > On Mon, May 07, 2018 at 09:22:47PM +0500, alp.ars...@xflowresearch.com > wrote: > > I mean OpenFlow flows that are persistent across OVS or system restarts. > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Monday, May 7, 2018 9:18 PM > > To: alp.ars...@xflowresearch.com > > Cc: disc...@openvswitch.org > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > 2.9 > > > > What do you mean by "persist the OVS flows"? I have a couple of > > guesses but I'd like to hear from you. > > > > On Mon, May 07, 2018 at 02:17:18AM +0500, > > alp.ars...@xflowresearch.com > > wrote: > > > Thank you Ben for the correction. I am running tests with > > > different scenarios to better understand what's happening inside the OVS-DPDK. > > > One thing that I would like to ask is, is there a way to persist > > > the OVS > > flows? > > > Can OVN help me do that? I don't have any virtual networks, just > > > in and out ports. > > > > > > -Original Message- > > > From: Ben Pfaff [mailto:b...@ovn.org] > > > Sent: Friday, May 4, 2018 11:29 AM > > > To: alp.ars...@xflowresearch.com > > > Cc: disc...@openvswitch.org > > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > > 2.9 > > > > > > It's mostly for historical reasons. > > > > > > We do try to document in ovs-vswitchd(8) that the user should not > > > manage datapaths themselves: > > > > > >ovs-vswitchd does all the necessary management of Open vSwitch > > >datapaths itself. Thus, external tools, such ovs-dpctl(8), are > > >not needed for managing datapaths in conjunction with > > >ovs-vswitchd, and their use to modify datapaths when ovs-vswitchd > > >is running can interfere with its operation. (ovs-dpctl may > > >still be useful for diagnostics.) > > > > > > I guess that the wording should be updated to reflect the "ovs-appctl"
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
Using a controller might yield better latency for adding and removing flows, but I'm a little surprised to hear that there's terribly slow latency when traffic is flowing. Usually, the main thread in OVS (which is the one that does OpenFlow table management) doesn't have much to do, even if there is a high traffic load. I don't have a general-purpose controller to recommend. There are several out there and I imagine that any one of them could be suitable for this project. On Mon, May 07, 2018 at 09:41:17PM +0500, alp.ars...@xflowresearch.com wrote: > What would you recommend as a controller, also would adding rules using a > controller be faster than "ovs-ofctl"? My application continuously add and > delete flows based on the traffic patterns, and it works fine if there is no > traffic flowing through the OVS, but as soon as I turn on the 40 G links, > the time to apply the rules starting hitting several seconds sometimes up to > 100s. It keeps increasing as the existing number of flows increase. > > Can an SDN controller solve this issue? Or is it an inherent OVS limitation > where it takes more time to add rules if more packets are going to the OF > classifier? > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Monday, May 7, 2018 9:36 PM > To: alp.ars...@xflowresearch.com > Cc: disc...@openvswitch.org > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > OVS doesn't have that built in. Usually we think of it as the > responsibility of the controller. > > On Mon, May 07, 2018 at 09:22:47PM +0500, alp.ars...@xflowresearch.com > wrote: > > I mean OpenFlow flows that are persistent across OVS or system restarts. > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Monday, May 7, 2018 9:18 PM > > To: alp.ars...@xflowresearch.com > > Cc: disc...@openvswitch.org > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > > > What do you mean by "persist the OVS flows"? I have a couple of > > guesses but I'd like to hear from you. > > > > On Mon, May 07, 2018 at 02:17:18AM +0500, alp.ars...@xflowresearch.com > > wrote: > > > Thank you Ben for the correction. I am running tests with different > > > scenarios to better understand what's happening inside the OVS-DPDK. > > > One thing that I would like to ask is, is there a way to persist the > > > OVS > > flows? > > > Can OVN help me do that? I don't have any virtual networks, just in > > > and out ports. > > > > > > -----Original Message- > > > From: Ben Pfaff [mailto:b...@ovn.org] > > > Sent: Friday, May 4, 2018 11:29 AM > > > To: alp.ars...@xflowresearch.com > > > Cc: disc...@openvswitch.org > > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > > 2.9 > > > > > > It's mostly for historical reasons. > > > > > > We do try to document in ovs-vswitchd(8) that the user should not > > > manage datapaths themselves: > > > > > >ovs-vswitchd does all the necessary management of Open vSwitch > > >datapaths itself. Thus, external tools, such ovs-dpctl(8), are > > >not needed for managing datapaths in conjunction with > > >ovs-vswitchd, and their use to modify datapaths when ovs-vswitchd > > >is running can interfere with its operation. (ovs-dpctl may > > >still be useful for diagnostics.) > > > > > > I guess that the wording should be updated to reflect the "ovs-appctl" > > > interface too. > > > > > > I sent a patch to improve the docs here: > > > https://patchwork.ozlabs.org/patch/908532/ > > > > > > On Thu, May 03, 2018 at 06:44:43PM +0500, > > > alp.ars...@xflowresearch.com > > > wrote: > > > > If "ovs-vswitchd" manages the data paths, why does it have a > > > > utility that lets me create more of them. And when I create them I > > > > cannot use them. I am stuck in a loop :) . > > > > > > > > -Original Message- > > > > From: Ben Pfaff [mailto:b...@ovn.org] > > > > Sent: Thursday, May 3, 2018 4:41 PM > > > > To: alp.ars...@xflowresearch.com > > > > Cc: disc...@openvswitch.org > > > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > > > 2.9 > > > > > > > > On Wed, May 02, 2018 at 10:02:04PM +0500, > > > > alp.ars...@xflowresearch.com > > >
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
On 05/07/2018 01:41 PM, alp.ars...@xflowresearch.com wrote: What would you recommend as a controller, also would adding rules using a controller be faster than "ovs-ofctl"? My application continuously add and delete flows based on the traffic patterns, and it works fine if there is no traffic flowing through the OVS, but as soon as I turn on the 40 G links, the time to apply the rules starting hitting several seconds sometimes up to 100s. It keeps increasing as the existing number of flows increase. Wouldn't it be better to run some sort of analysis to determine at what point your flow insertions start to fall behind? Are you setting up many short lived flows? Do you have jumbo flows? Or some mixture? Then figure out how many flows adjustments you need from testing. Linux should be able to handle 10G. 40G, without a bunch of tuning and analysis and distribution, could be challenging. And depends upon what stuff is being off-loaded, and what is calculated locally. At that point, you have choices: Ryu: python based controller with an established support base, but might be slow for your needs Some Java based engines: compiled and used by some bigger boys Come C++ engines: fast code, and maybe fast insertions (c++ and java would send messages which might bypass some of the ovs-ofctl parsing) Some people diddle with certain functions directly (see past archives for several mentions) Or maybe use P4 or eBPF to handle your packet handling? Can an SDN controller solve this issue? Or is it an inherent OVS limitation where it takes more time to add rules if more packets are going to the OF classifier? -- Raymond Burkholder r...@oneunified.net https://blog.raymond.burkholder.net -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
What would you recommend as a controller, also would adding rules using a controller be faster than "ovs-ofctl"? My application continuously add and delete flows based on the traffic patterns, and it works fine if there is no traffic flowing through the OVS, but as soon as I turn on the 40 G links, the time to apply the rules starting hitting several seconds sometimes up to 100s. It keeps increasing as the existing number of flows increase. Can an SDN controller solve this issue? Or is it an inherent OVS limitation where it takes more time to add rules if more packets are going to the OF classifier? -Original Message- From: Ben Pfaff [mailto:b...@ovn.org] Sent: Monday, May 7, 2018 9:36 PM To: alp.ars...@xflowresearch.com Cc: disc...@openvswitch.org Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 OVS doesn't have that built in. Usually we think of it as the responsibility of the controller. On Mon, May 07, 2018 at 09:22:47PM +0500, alp.ars...@xflowresearch.com wrote: > I mean OpenFlow flows that are persistent across OVS or system restarts. > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Monday, May 7, 2018 9:18 PM > To: alp.ars...@xflowresearch.com > Cc: disc...@openvswitch.org > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > What do you mean by "persist the OVS flows"? I have a couple of > guesses but I'd like to hear from you. > > On Mon, May 07, 2018 at 02:17:18AM +0500, alp.ars...@xflowresearch.com > wrote: > > Thank you Ben for the correction. I am running tests with different > > scenarios to better understand what's happening inside the OVS-DPDK. > > One thing that I would like to ask is, is there a way to persist the > > OVS > flows? > > Can OVN help me do that? I don't have any virtual networks, just in > > and out ports. > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Friday, May 4, 2018 11:29 AM > > To: alp.ars...@xflowresearch.com > > Cc: disc...@openvswitch.org > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > 2.9 > > > > It's mostly for historical reasons. > > > > We do try to document in ovs-vswitchd(8) that the user should not > > manage datapaths themselves: > > > >ovs-vswitchd does all the necessary management of Open vSwitch > >datapaths itself. Thus, external tools, such ovs-dpctl(8), are > >not needed for managing datapaths in conjunction with > >ovs-vswitchd, and their use to modify datapaths when ovs-vswitchd > >is running can interfere with its operation. (ovs-dpctl may > >still be useful for diagnostics.) > > > > I guess that the wording should be updated to reflect the "ovs-appctl" > > interface too. > > > > I sent a patch to improve the docs here: > > https://patchwork.ozlabs.org/patch/908532/ > > > > On Thu, May 03, 2018 at 06:44:43PM +0500, > > alp.ars...@xflowresearch.com > > wrote: > > > If "ovs-vswitchd" manages the data paths, why does it have a > > > utility that lets me create more of them. And when I create them I > > > cannot use them. I am stuck in a loop :) . > > > > > > -Original Message- > > > From: Ben Pfaff [mailto:b...@ovn.org] > > > Sent: Thursday, May 3, 2018 4:41 PM > > > To: alp.ars...@xflowresearch.com > > > Cc: disc...@openvswitch.org > > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > > 2.9 > > > > > > On Wed, May 02, 2018 at 10:02:04PM +0500, > > > alp.ars...@xflowresearch.com > > > wrote: > > > > I am trying to create multiple dpdk-netdev based data paths with > > > > OVS > > > > 2.9 and DPDK 16.11 running on CentOS 7.4. I am able to create > > > > multiple data paths using "ovs-appctl dpctl/add-dp netdev@netdev1" > > > > and I can see a new data path created with "ovs-appctl dpctl/show". > > > > However I cannot add any interfaces (dpdk or otherwise), and I > > > > cannot set this data path as datapath_type to any bridge. > > > > > > That's not useful or a good idea. ovs-vswitchd manages datapaths > itself. > > > Adding and removing them yourself will not help. > > > > > > > Just a precap to why I am trying to do this, I am working with a > > > > lot of OVS OpenFlow rules (around 0.5 million) matching layer 3 > > > > and layer > > > > 4 fields. The incoming traff
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
OVS doesn't have that built in. Usually we think of it as the responsibility of the controller. On Mon, May 07, 2018 at 09:22:47PM +0500, alp.ars...@xflowresearch.com wrote: > I mean OpenFlow flows that are persistent across OVS or system restarts. > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Monday, May 7, 2018 9:18 PM > To: alp.ars...@xflowresearch.com > Cc: disc...@openvswitch.org > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > What do you mean by "persist the OVS flows"? I have a couple of guesses but > I'd like to hear from you. > > On Mon, May 07, 2018 at 02:17:18AM +0500, alp.ars...@xflowresearch.com > wrote: > > Thank you Ben for the correction. I am running tests with different > > scenarios to better understand what's happening inside the OVS-DPDK. > > One thing that I would like to ask is, is there a way to persist the OVS > flows? > > Can OVN help me do that? I don't have any virtual networks, just in > > and out ports. > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Friday, May 4, 2018 11:29 AM > > To: alp.ars...@xflowresearch.com > > Cc: disc...@openvswitch.org > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > > > It's mostly for historical reasons. > > > > We do try to document in ovs-vswitchd(8) that the user should not > > manage datapaths themselves: > > > >ovs-vswitchd does all the necessary management of Open vSwitch > >datapaths itself. Thus, external tools, such ovs-dpctl(8), are > >not needed for managing datapaths in conjunction with > >ovs-vswitchd, and their use to modify datapaths when ovs-vswitchd > >is running can interfere with its operation. (ovs-dpctl may > >still be useful for diagnostics.) > > > > I guess that the wording should be updated to reflect the "ovs-appctl" > > interface too. > > > > I sent a patch to improve the docs here: > > https://patchwork.ozlabs.org/patch/908532/ > > > > On Thu, May 03, 2018 at 06:44:43PM +0500, alp.ars...@xflowresearch.com > > wrote: > > > If "ovs-vswitchd" manages the data paths, why does it have a utility > > > that lets me create more of them. And when I create them I cannot > > > use them. I am stuck in a loop :) . > > > > > > -Original Message- > > > From: Ben Pfaff [mailto:b...@ovn.org] > > > Sent: Thursday, May 3, 2018 4:41 PM > > > To: alp.ars...@xflowresearch.com > > > Cc: disc...@openvswitch.org > > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > > 2.9 > > > > > > On Wed, May 02, 2018 at 10:02:04PM +0500, > > > alp.ars...@xflowresearch.com > > > wrote: > > > > I am trying to create multiple dpdk-netdev based data paths with > > > > OVS > > > > 2.9 and DPDK 16.11 running on CentOS 7.4. I am able to create > > > > multiple data paths using "ovs-appctl dpctl/add-dp netdev@netdev1" > > > > and I can see a new data path created with "ovs-appctl dpctl/show". > > > > However I cannot add any interfaces (dpdk or otherwise), and I > > > > cannot set this data path as datapath_type to any bridge. > > > > > > That's not useful or a good idea. ovs-vswitchd manages datapaths > itself. > > > Adding and removing them yourself will not help. > > > > > > > Just a precap to why I am trying to do this, I am working with a > > > > lot of OVS OpenFlow rules (around 0.5 million) matching layer 3 > > > > and layer > > > > 4 fields. The incoming traffic is more than 40G (4 x10G Intel > > > > x520s), and has multiple parallel flows (over a million IPs). With > > > > this the OVS performance decreases and each port is forwarding > > > > only around 250 Mb/s. I am using multiple RX queues (4-6), with > > > > single RX queue it drops to 70 Mb/s. Now if I shutdown three 10G > > > > interfaces, an interesting thing happen, and OVS starts forwarding > > > > over 7Gb/s for that single interface. That got me thinking, maybe > > > > the reason for low performance is 40 G traffic hitting a single > > > > bridges flow tables, how about creating multiple bridges with > > > > multiple flow tables. With this setup the situation remained same, > > > > and now the only common thing between the > > > &
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
I mean OpenFlow flows that are persistent across OVS or system restarts. -Original Message- From: Ben Pfaff [mailto:b...@ovn.org] Sent: Monday, May 7, 2018 9:18 PM To: alp.ars...@xflowresearch.com Cc: disc...@openvswitch.org Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 What do you mean by "persist the OVS flows"? I have a couple of guesses but I'd like to hear from you. On Mon, May 07, 2018 at 02:17:18AM +0500, alp.ars...@xflowresearch.com wrote: > Thank you Ben for the correction. I am running tests with different > scenarios to better understand what's happening inside the OVS-DPDK. > One thing that I would like to ask is, is there a way to persist the OVS flows? > Can OVN help me do that? I don't have any virtual networks, just in > and out ports. > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Friday, May 4, 2018 11:29 AM > To: alp.ars...@xflowresearch.com > Cc: disc...@openvswitch.org > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > It's mostly for historical reasons. > > We do try to document in ovs-vswitchd(8) that the user should not > manage datapaths themselves: > >ovs-vswitchd does all the necessary management of Open vSwitch >datapaths itself. Thus, external tools, such ovs-dpctl(8), are >not needed for managing datapaths in conjunction with >ovs-vswitchd, and their use to modify datapaths when ovs-vswitchd >is running can interfere with its operation. (ovs-dpctl may >still be useful for diagnostics.) > > I guess that the wording should be updated to reflect the "ovs-appctl" > interface too. > > I sent a patch to improve the docs here: > https://patchwork.ozlabs.org/patch/908532/ > > On Thu, May 03, 2018 at 06:44:43PM +0500, alp.ars...@xflowresearch.com > wrote: > > If "ovs-vswitchd" manages the data paths, why does it have a utility > > that lets me create more of them. And when I create them I cannot > > use them. I am stuck in a loop :) . > > > > -Original Message- > > From: Ben Pfaff [mailto:b...@ovn.org] > > Sent: Thursday, May 3, 2018 4:41 PM > > To: alp.ars...@xflowresearch.com > > Cc: disc...@openvswitch.org > > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS > > 2.9 > > > > On Wed, May 02, 2018 at 10:02:04PM +0500, > > alp.ars...@xflowresearch.com > > wrote: > > > I am trying to create multiple dpdk-netdev based data paths with > > > OVS > > > 2.9 and DPDK 16.11 running on CentOS 7.4. I am able to create > > > multiple data paths using "ovs-appctl dpctl/add-dp netdev@netdev1" > > > and I can see a new data path created with "ovs-appctl dpctl/show". > > > However I cannot add any interfaces (dpdk or otherwise), and I > > > cannot set this data path as datapath_type to any bridge. > > > > That's not useful or a good idea. ovs-vswitchd manages datapaths itself. > > Adding and removing them yourself will not help. > > > > > Just a precap to why I am trying to do this, I am working with a > > > lot of OVS OpenFlow rules (around 0.5 million) matching layer 3 > > > and layer > > > 4 fields. The incoming traffic is more than 40G (4 x10G Intel > > > x520s), and has multiple parallel flows (over a million IPs). With > > > this the OVS performance decreases and each port is forwarding > > > only around 250 Mb/s. I am using multiple RX queues (4-6), with > > > single RX queue it drops to 70 Mb/s. Now if I shutdown three 10G > > > interfaces, an interesting thing happen, and OVS starts forwarding > > > over 7Gb/s for that single interface. That got me thinking, maybe > > > the reason for low performance is 40 G traffic hitting a single > > > bridges flow tables, how about creating multiple bridges with > > > multiple flow tables. With this setup the situation remained same, > > > and now the only common thing between the > > > 4 interfaces is the data path. They are not sharing anything else. > > > They are polled by dedicated vCPUs, and they are in different tables. > > > > > > > > > > > > Can anyone explain this bizarre scenario of why the OVS is able to > > > forward more traffic over single interface polled by 6 vCPUs, > > > compared to 4 interfaces polled by 24 vCPUs. > > > > > > Also is there a way to create multiple data paths and remove this > > > dependency also. > > > > You can create multiple bridges with "ovs-vsctl add-br". OVS > > doesn't use multiple datapaths. > > > > Maybe someone who understands the DPDK port better can suggest some > > reason for the performance characteristics that you see. > > > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
Yes, using the pmd-stats-show command I can see that about half the traffic has to go to the OF Classifier, and if I watch the "ovs-appctl dpctl/show" the number of flows in the dpctl is changing very rapidly, is there a way to increase the time flows stick in the dpctl? -Original Message- From: O'Reilly, Darragh [mailto:darragh.orei...@hpe.com] Sent: Thursday, May 3, 2018 10:05 PM To: alp.ars...@xflowresearch.com; disc...@openvswitch.org Subject: RE: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > -Original Message- > From: alp.ars...@xflowresearch.com > [mailto:alp.ars...@xflowresearch.com] > > Number of flows and incoming traffic distribution is really high. > Number of rules is +0.5 million and possible number of IP of incoming traffic is also in millions. > But right now I am sending same pcap file to all interfaces using > PKTGEN. There is no slow path as it is OVS-DPDK, everything in userspace. > By slow path I mean the ofproto classifier: https://software.intel.com/en-us/articles/ovs-dpdk-datapath-classifier-part- 2 ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
Thank you Ben for the correction. I am running tests with different scenarios to better understand what's happening inside the OVS-DPDK. One thing that I would like to ask is, is there a way to persist the OVS flows? Can OVN help me do that? I don't have any virtual networks, just in and out ports. -Original Message- From: Ben Pfaff [mailto:b...@ovn.org] Sent: Friday, May 4, 2018 11:29 AM To: alp.ars...@xflowresearch.com Cc: disc...@openvswitch.org Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 It's mostly for historical reasons. We do try to document in ovs-vswitchd(8) that the user should not manage datapaths themselves: ovs-vswitchd does all the necessary management of Open vSwitch datapaths itself. Thus, external tools, such ovs-dpctl(8), are not needed for managing datapaths in conjunction with ovs-vswitchd, and their use to modify datapaths when ovs-vswitchd is running can interfere with its operation. (ovs-dpctl may still be useful for diagnostics.) I guess that the wording should be updated to reflect the "ovs-appctl" interface too. I sent a patch to improve the docs here: https://patchwork.ozlabs.org/patch/908532/ On Thu, May 03, 2018 at 06:44:43PM +0500, alp.ars...@xflowresearch.com wrote: > If "ovs-vswitchd" manages the data paths, why does it have a utility > that lets me create more of them. And when I create them I cannot use > them. I am stuck in a loop :) . > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: Thursday, May 3, 2018 4:41 PM > To: alp.ars...@xflowresearch.com > Cc: disc...@openvswitch.org > Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > > On Wed, May 02, 2018 at 10:02:04PM +0500, alp.ars...@xflowresearch.com > wrote: > > I am trying to create multiple dpdk-netdev based data paths with OVS > > 2.9 and DPDK 16.11 running on CentOS 7.4. I am able to create > > multiple data paths using "ovs-appctl dpctl/add-dp netdev@netdev1" > > and I can see a new data path created with "ovs-appctl dpctl/show". > > However I cannot add any interfaces (dpdk or otherwise), and I > > cannot set this data path as datapath_type to any bridge. > > That's not useful or a good idea. ovs-vswitchd manages datapaths itself. > Adding and removing them yourself will not help. > > > Just a precap to why I am trying to do this, I am working with a lot > > of OVS OpenFlow rules (around 0.5 million) matching layer 3 and > > layer > > 4 fields. The incoming traffic is more than 40G (4 x10G Intel > > x520s), and has multiple parallel flows (over a million IPs). With > > this the OVS performance decreases and each port is forwarding only > > around 250 Mb/s. I am using multiple RX queues (4-6), with single RX > > queue it drops to 70 Mb/s. Now if I shutdown three 10G interfaces, > > an interesting thing happen, and OVS starts forwarding over 7Gb/s > > for that single interface. That got me thinking, maybe the reason > > for low performance is 40 G traffic hitting a single bridges flow > > tables, how about creating multiple bridges with multiple flow > > tables. With this setup the situation remained same, and now the > > only common thing between the > > 4 interfaces is the data path. They are not sharing anything else. > > They are polled by dedicated vCPUs, and they are in different tables. > > > > > > > > Can anyone explain this bizarre scenario of why the OVS is able to > > forward more traffic over single interface polled by 6 vCPUs, > > compared to 4 interfaces polled by 24 vCPUs. > > > > Also is there a way to create multiple data paths and remove this > > dependency also. > > You can create multiple bridges with "ovs-vsctl add-br". OVS doesn't > use multiple datapaths. > > Maybe someone who understands the DPDK port better can suggest some > reason for the performance characteristics that you see. > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
> -Original Message- > From: alp.ars...@xflowresearch.com [mailto:alp.ars...@xflowresearch.com] > > Number of flows and incoming traffic distribution is really high. Number of > rules > is +0.5 million and possible number of IP of incoming traffic is also in > millions. > But right now I am sending same pcap file to all interfaces using PKTGEN. > There > is no slow path as it is OVS-DPDK, everything in userspace. > By slow path I mean the ofproto classifier: https://software.intel.com/en-us/articles/ovs-dpdk-datapath-classifier-part-2 ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
Number of flows and incoming traffic distribution is really high. Number of rules is +0.5 million and possible number of IP of incoming traffic is also in millions. But right now I am sending same pcap file to all interfaces using PKTGEN. There is no slow path as it is OVS-DPDK, everything in userspace. -Original Message- From: O'Reilly, Darragh [mailto:darragh.orei...@hpe.com] Sent: Thursday, May 3, 2018 8:05 PM To: alp.ars...@xflowresearch.com; disc...@openvswitch.org Subject: RE: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 > -Original Message- > From: alp.ars...@xflowresearch.com > [mailto:alp.ars...@xflowresearch.com] > > Enabling disabling EMC has no effect on this scenario. As far as I > know there is one EMC per PMD thread, so the interfaces have their own > EMC's, the bigger question is why does traffic on one interface effect > the performance of the other? Are they sharing anything? The only > thing I can think of is the datapath and the megaflow table, and I am > looking for some way to do that, is this doesn't work my only other > option is to have 4 VMs with pass-through interfaces and run OVS-DPDK inside VMs. > Could it be the nature of the test traffic, flows and total rate of received packets with 4 NICs? Is the slow path thread ovs-vswitchd maxed out? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
If "ovs-vswitchd" manages the data paths, why does it have a utility that lets me create more of them. And when I create them I cannot use them. I am stuck in a loop :) . -Original Message- From: Ben Pfaff [mailto:b...@ovn.org] Sent: Thursday, May 3, 2018 4:41 PM To: alp.ars...@xflowresearch.com Cc: disc...@openvswitch.org Subject: Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 On Wed, May 02, 2018 at 10:02:04PM +0500, alp.ars...@xflowresearch.com wrote: > I am trying to create multiple dpdk-netdev based data paths with OVS > 2.9 and DPDK 16.11 running on CentOS 7.4. I am able to create multiple > data paths using "ovs-appctl dpctl/add-dp netdev@netdev1" and I can > see a new data path created with "ovs-appctl dpctl/show". However I > cannot add any interfaces (dpdk or otherwise), and I cannot set this > data path as datapath_type to any bridge. That's not useful or a good idea. ovs-vswitchd manages datapaths itself. Adding and removing them yourself will not help. > Just a precap to why I am trying to do this, I am working with a lot > of OVS OpenFlow rules (around 0.5 million) matching layer 3 and layer > 4 fields. The incoming traffic is more than 40G (4 x10G Intel x520s), > and has multiple parallel flows (over a million IPs). With this the > OVS performance decreases and each port is forwarding only around 250 > Mb/s. I am using multiple RX queues (4-6), with single RX queue it > drops to 70 Mb/s. Now if I shutdown three 10G interfaces, an > interesting thing happen, and OVS starts forwarding over 7Gb/s for > that single interface. That got me thinking, maybe the reason for low > performance is 40 G traffic hitting a single bridges flow tables, how > about creating multiple bridges with multiple flow tables. With this > setup the situation remained same, and now the only common thing > between the > 4 interfaces is the data path. They are not sharing anything else. > They are polled by dedicated vCPUs, and they are in different tables. > > > > Can anyone explain this bizarre scenario of why the OVS is able to > forward more traffic over single interface polled by 6 vCPUs, compared > to 4 interfaces polled by 24 vCPUs. > > Also is there a way to create multiple data paths and remove this > dependency also. You can create multiple bridges with "ovs-vsctl add-br". OVS doesn't use multiple datapaths. Maybe someone who understands the DPDK port better can suggest some reason for the performance characteristics that you see. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
Enabling disabling EMC has no effect on this scenario. As far as I know there is one EMC per PMD thread, so the interfaces have their own EMC's, the bigger question is why does traffic on one interface effect the performance of the other? Are they sharing anything? The only thing I can think of is the datapath and the megaflow table, and I am looking for some way to do that, is this doesn't work my only other option is to have 4 VMs with pass-through interfaces and run OVS-DPDK inside VMs. -Original Message- From: O'Reilly, Darragh [mailto:darragh.orei...@hpe.com] Sent: Thursday, May 3, 2018 5:49 PM To: alp.ars...@xflowresearch.com; disc...@openvswitch.org Subject: RE: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9 On Wed, May 02, 2018 at 10:02:04PM +0500, alp.ars...@xflowresearch.com wrote: > Can anyone explain this bizarre scenario of why the OVS is able to > forward more traffic over single interface polled by 6 vCPUs, compared > to 4 interfaces polled by 24 vCPUs. Not really, but I would look at the cache stats: ovs-appctl dpif-netdev/pmd-stats-show ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
> -Original Message- > From: alp.ars...@xflowresearch.com [mailto:alp.ars...@xflowresearch.com] > > Enabling disabling EMC has no effect on this scenario. As far as I know there > is > one EMC per PMD thread, so the interfaces have their own EMC's, the bigger > question is why does traffic on one interface effect the performance of the > other? Are they sharing anything? The only thing I can think of is the > datapath > and the megaflow table, and I am looking for some way to do that, is this > doesn't work my only other option is to have 4 VMs with pass-through > interfaces > and run OVS-DPDK inside VMs. > Could it be the nature of the test traffic, flows and total rate of received packets with 4 NICs? Is the slow path thread ovs-vswitchd maxed out? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
On Wed, May 02, 2018 at 10:02:04PM +0500, alp.ars...@xflowresearch.com wrote: > Can anyone explain this bizarre scenario of why the OVS is able to > forward more traffic over single interface polled by 6 vCPUs, compared > to 4 interfaces polled by 24 vCPUs. Not really, but I would look at the cache stats: ovs-appctl dpif-netdev/pmd-stats-show ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
On Wed, May 02, 2018 at 10:02:04PM +0500, alp.ars...@xflowresearch.com wrote: > I am trying to create multiple dpdk-netdev based data paths with OVS 2.9 and > DPDK 16.11 running on CentOS 7.4. I am able to create multiple data paths > using "ovs-appctl dpctl/add-dp netdev@netdev1" and I can see a new data path > created with "ovs-appctl dpctl/show". However I cannot add any interfaces > (dpdk or otherwise), and I cannot set this data path as datapath_type to any > bridge. That's not useful or a good idea. ovs-vswitchd manages datapaths itself. Adding and removing them yourself will not help. > Just a precap to why I am trying to do this, I am working with a lot of OVS > OpenFlow rules (around 0.5 million) matching layer 3 and layer 4 fields. The > incoming traffic is more than 40G (4 x10G Intel x520s), and has multiple > parallel flows (over a million IPs). With this the OVS performance decreases > and each port is forwarding only around 250 Mb/s. I am using multiple RX > queues (4-6), with single RX queue it drops to 70 Mb/s. Now if I shutdown > three 10G interfaces, an interesting thing happen, and OVS starts forwarding > over 7Gb/s for that single interface. That got me thinking, maybe the reason > for low performance is 40 G traffic hitting a single bridges flow tables, > how about creating multiple bridges with multiple flow tables. With this > setup the situation remained same, and now the only common thing between the > 4 interfaces is the data path. They are not sharing anything else. They are > polled by dedicated vCPUs, and they are in different tables. > > > > Can anyone explain this bizarre scenario of why the OVS is able to forward > more traffic over single interface polled by 6 vCPUs, compared to 4 > interfaces polled by 24 vCPUs. > > Also is there a way to create multiple data paths and remove this dependency > also. You can create multiple bridges with "ovs-vsctl add-br". OVS doesn't use multiple datapaths. Maybe someone who understands the DPDK port better can suggest some reason for the performance characteristics that you see. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] Multiple dpdk-netdev datapath with OVS 2.9
Dear All, I am trying to create multiple dpdk-netdev based data paths with OVS 2.9 and DPDK 16.11 running on CentOS 7.4. I am able to create multiple data paths using "ovs-appctl dpctl/add-dp netdev@netdev1" and I can see a new data path created with "ovs-appctl dpctl/show". However I cannot add any interfaces (dpdk or otherwise), and I cannot set this data path as datapath_type to any bridge. Just a precap to why I am trying to do this, I am working with a lot of OVS OpenFlow rules (around 0.5 million) matching layer 3 and layer 4 fields. The incoming traffic is more than 40G (4 x10G Intel x520s), and has multiple parallel flows (over a million IPs). With this the OVS performance decreases and each port is forwarding only around 250 Mb/s. I am using multiple RX queues (4-6), with single RX queue it drops to 70 Mb/s. Now if I shutdown three 10G interfaces, an interesting thing happen, and OVS starts forwarding over 7Gb/s for that single interface. That got me thinking, maybe the reason for low performance is 40 G traffic hitting a single bridges flow tables, how about creating multiple bridges with multiple flow tables. With this setup the situation remained same, and now the only common thing between the 4 interfaces is the data path. They are not sharing anything else. They are polled by dedicated vCPUs, and they are in different tables. Can anyone explain this bizarre scenario of why the OVS is able to forward more traffic over single interface polled by 6 vCPUs, compared to 4 interfaces polled by 24 vCPUs. Also is there a way to create multiple data paths and remove this dependency also. Regards, Alp Arslan ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
