Hi All, I've posted v2 with Darrell's recommendations included.
It doesn't appear in ovs-dev but it's on patchwork - https://patchwork.ozlabs.org/patch/745195/ so I'm not sure what's going on there. The same thing happened with another patch I sent earlier. Cheers, Billy. > -----Original Message----- > From: [email protected] [mailto:ovs-dev- > [email protected]] On Behalf Of O Mahony, Billy > Sent: Thursday, March 30, 2017 10:11 AM > To: Darrell Ball <[email protected]>; Jan Scheurich > <[email protected]>; Ilya Maximets <[email protected]>; > [email protected] > Subject: Re: [ovs-dev] dpif-netdev: Assign ports to pmds on non-local numa > node. > > Hi Darrell, > > > -----Original Message----- > > From: Darrell Ball [mailto:[email protected]] > > Sent: Wednesday, March 29, 2017 6:35 PM > > To: O Mahony, Billy <[email protected]>; Jan Scheurich > > <[email protected]>; Ilya Maximets > <[email protected]>; > > [email protected] > > Subject: Re: [ovs-dev] dpif-netdev: Assign ports to pmds on non-local > > numa node. > > > > I see there is warning log added for the non-local case. > > I wonder if during setup, there will be no issues with basic connectivity > seen. > > > > Then some long time later at high PPS usage, unexpected low > > performance will be seen. > > I am not sure whether the log will be immediately consulted to check > > for this, as the reasons for low PPS performance would likely be > > elsewhere; this is pretty subtle. > > > > I think there needs to be documentation updates associated with this > > change explaining this. > > Ideally the warning would be referenced in the documentation as clear > > cross reference. > [[BO'M]] That's a good idea. I'll submit a v2 shortly. > > > > Thanks > > Darrell > > > > > > On 3/9/17, 7:16 AM, "[email protected] on behalf of O > > Mahony, Billy" <[email protected] on behalf of > > [email protected]> wrote: > > > > Hi Jan, > > > > > > > > Thanks. That is another valid point if you don't - currently if > > you have dual- socket (numa) system then you have to spend at least 2 > > cores on your ovs- dpdk vswitch. With this patch it will allow such > > systems to operate on just one core (with reduced switching capacity of > course). > > > > > > > > I'd be interested to hear any other comments from the community on > > this patch. > > > > > > > > Regards, > > > > /Billy > > > > > > > > > -----Original Message----- > > > > > From: Jan Scheurich [mailto:[email protected]] > > > > > Sent: Monday, March 6, 2017 8:25 AM > > > > > To: O Mahony, Billy <[email protected]>; Ilya Maximets > > > > > <[email protected]>; [email protected] > > > > > Subject: RE: [ovs-dev] dpif-netdev: Assign ports to pmds on > > non-local numa > > > > > node. > > > > > > > > > > I support Billy's point here. There are a number of > > constraints/trade-offs that > > > > > can lead to an OVS deployments in OpenStack with e.g. a single PMD. > > > > > Without this change, a dual socket system is effectively > > rendered use- less. At > > > > > least one socket is lost for running VMs. > > > > > > > > > > BR, Jan > > > > > > > > > > > -----Original Message----- > > > > > > From: [email protected] > > > > > > [mailto:[email protected]] On Behalf Of O > > Mahony, Billy > > > > > > Sent: Tuesday, 28 February, 2017 14:59 > > > > > > To: Ilya Maximets <[email protected]>; > > [email protected] > > > > > > Subject: Re: [ovs-dev] dpif-netdev: Assign ports to pmds on > > non-local > > > > > numa node. > > > > > > > > > > > > Hi Ilya, > > > > > > > > > > > > Comments below. > > > > > > > > > > > > BR, > > > > > > Billy. > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Ilya Maximets [mailto:[email protected]] > > > > > > > Sent: Tuesday, February 28, 2017 12:49 PM > > > > > > > To: O Mahony, Billy <[email protected]>; > > [email protected] > > > > > > > Cc: Daniele Di Proietto <[email protected]> > > > > > > > Subject: Re: [ovs-dev] dpif-netdev: Assign ports to pmds on > > > > > > > non-local numa node. > > > > > > > > > > > > > > On 28.02.2017 14:43, O Mahony, Billy wrote: > > > > > > > > Hi Ilya, > > > > > > > > > > > > > > > > Thanks for the quick response. You make some good points. > > > > > > > > > > > > > > > > It's important to point out that local pmds are still > > chosen when > > > > > available. > > > > > > > > The change only operates to avoid totally non-operational/ > > > > > > > > non-polled > > > > > > > ports. > > > > > > > > > > > > > > > > This is something we came across when deploying > > DPDK-enabled OVS > > > > > > > > in OpenStack environments (OPNFV project). Where we had > > remote > > > > > > > > (both physically and administratively) multi-node labs > > already > > > > > > > > wired up and would have much preferred to have sub-optimal > > > > > > > > operation that a non- > > > > > > > operational OpenStack environment. > > > > > > > > > > > > > > > > Some further comments below. > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Billy > > > > > > > > > > > > > > > >> -----Original Message----- > > > > > > > >> From: Ilya Maximets [mailto:[email protected]] > > > > > > > >> Sent: Tuesday, February 28, 2017 11:21 AM > > > > > > > >> To: O Mahony, Billy <[email protected]>; > > > > > > > >> [email protected] > > > > > > > >> Cc: Daniele Di Proietto <[email protected]> > > > > > > > >> Subject: Re: [ovs-dev] dpif-netdev: Assign ports to pmds > > on > > > > > > > >> non-local numa node. > > > > > > > >> > > > > > > > >> Hello. > > > > > > > >> > > > > > > > >> On 28.02.2017 13:12, Billy O'Mahony wrote: > > > > > > > >>> From: billyom <[email protected]> > > > > > > > >>> > > > > > > > >>> Previously if there is no available (non-isolated) pmd > > on the > > > > > > > >>> numa node for a port then the port is not polled at all. > > This > > > > > > > >>> can result in a non-operational system until such time > > as nics > > > > > > > >>> are physically repositioned. > > > > > > > >> > > > > > > > >> Why you can't just reconfigure your pmd-cpu-mask after NICs' > > > > > > > repositioning? > > > > > > > > [[BO'M]] The idea is to avoid having to repositioning NICs > > as they > > > > > > > > may be in remote data-centers administered by other > > organisations. > > > > > > > > Also this can be related to multi-node clusters where it > > won't be > > > > > > > > just one NIC on one system that needs to be moved but > > several > > > > > > > > NICS/nodes > > > > > > > affected. > > > > > > > > > > > > > > > > > > > > > > > >> Maybe you can use pmd-rxq-affinity to assign port on > > another NUMA > > > > > > > node? > > > > > > > > [[BO'M]] The low-performance assignment will only occur if > > there > > > > > > > > is no available PMD on the local NUMA node. Ie if possible > > the > > > > > > > > normal highly performant assignment is made. It is only > > when it is > > > > > > > > a choice between lower performance and total > > non-performance that > > > > > > > > the lesser of two evils > > > > > > > is chosen. > > > > > > > > > > > > > > Maybe you can include at least one core from each node in > > pmd-cpu- > > > > > mask? > > > > > > > > > > > > [[BO'M]] Ideally everyone using and developing the many > > OpenStack > > > > > > deployment tools would be aware of the optimal settings but we > > can > > > > > > expect that this will not the case in many situations. Poor > > > > > > configurations will occur. A warning message is included in > > the VLOG > > > > > > in this case to give users a hint that performance will not be > > optimal but > > > > > without imposing the penalty of a broken system - which is a > > heavy penalty if > > > > > you don’t even have direct access to the pmd-cpu-mask as you are > > driving > > > > > the ovs configuration via an OpenStack installation tool such as > > Fuel, RDO, > > > > > etc. > > > > > > > > > > > > > > > > > > > > >> The main concern here is that this 'remote' port will > > degrade > > > > > > > >> performance of other ports served by chosen PMD thread > > > > > significantly. > > > > > > > > [[BO'M]] That is a good point that there are second-order > > > > > > > > consequences on > > > > > > > other ports. > > > > > > > > But certainly with a many cloud systems even if one port > > is > > > > > > > > non-operational it means the entire node is effectively > > down - for > > > > > > > > instance an OpenStack compute node with a non-working > > provider > > > > > > > network > > > > > > > > for it's tenant VMs is useless even though the > > administration and > > > > > > > > control > > > > > > > networks are still working. > > > > > > > >> > > > > > > > >> Best regards, Ilya Maximets. > > > > > > _______________________________________________ > > > > > > dev mailing list > > > > > > [email protected] > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__mail.openvswitch.org_mailman_listinfo_ovs- > > 2Ddev&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > > > uZnsw&m=GNRRfh6vBdD0g81qjkYyoE4hWv80KC2GRbQro6HR_go&s=0nEw3l > > SrDPhqOGsgc7cUQhvvyh90VzJgqRbMXaEG-B8&e= > > > > _______________________________________________ > > dev mailing list > > [email protected] > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__mail.openvswitch.org_mailman_listinfo_ovs- > > 2Ddev&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > > > uZnsw&m=GNRRfh6vBdD0g81qjkYyoE4hWv80KC2GRbQro6HR_go&s=0nEw3l > > SrDPhqOGsgc7cUQhvvyh90VzJgqRbMXaEG-B8&e= > > > > _______________________________________________ > dev mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
