> -----Original Message----- > From: William Tu <[email protected]> > Sent: Thursday, May 21, 2020 11:30 PM > To: Van Haaren, Harry <[email protected]> > Cc: [email protected]; [email protected] > Subject: Re: [ovs-dev] [PATCH v2 5/5] dpif-lookup: add avx512 gather > implementation > > > > ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev > > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3 > > > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true > > > ovs-ofctl add-flow br0 'actions=drop' > > > ovs-appctl dpif-netdev/subtable-lookup-set avx512_gather 5 > > > ovs-vsctl add-port br0 tg0 -- set int tg0 type=dpdk \ > > > options:dpdk- > > > devargs=vdev:net_pcap0,rx_pcap=/root/ovs/p0.pcap,infinite_rx=1 > > > > I use Ether/VLAN/IPv4 to achieve a subtable with (4,1), is that the same as > you? > > Just trying to remove variables between our setups. > > > btw I have only one OpenFlow rule, 'actions=drop' > The pcap file as input is a random one I just capture in my machine's > interface > root@instance-3:~/ovs# tcpdump -n -r p0.pcap | wc -l > reading from file p0.pcap, link-type EN10MB (Ethernet) > 22 > root@instance-3:~/ovs# tcpdump -n -r p0.pcap
Hi William, Thanks for the info - I was away on a longer weekend, back now! <snip lots of pcap & details> > > > 2020-05-20T13:49:26.648Z|00057|netdev_dpdk|INFO|Port 0: > 02:70:63:61:70:00 > > > 2020-05-20T13:49:26.648Z|00058|dpif_netdev|INFO|Core 0 on numa node 0 > > > assigned port 'tg0' rx queue 0 (measured processing cycles 0). > > > 2020-05-20T13:49:26.648Z|00059|bridge|INFO|bridge br0: added interface > > > tg0 on port 1 > > > 2020-05-20T13:49:26.648Z|00001|ofproto_dpif_upcall(pmd- > > > c00/id:9)|WARN|upcall_cb > > > failure: ukey installation fails > > > 2020-05-20T13:49:27.562Z|00002|dpif_netdev(pmd- > c00/id:9)|INFO|reprobing > > > sub func, 4 1 > > > > Aha! This shows somethings going wrong - there should not be any ukey- > install fails! > > > > This also explains why your logs (as per follow-up email in thread) have a > > high > upcall count/sec, > > the installed flow isn't being hit when matched. I'm not sure what the root > cause of these > > ukey-installation fails are - but this is what we need to investigate :) > > > > Understanding the traffic, and attempting to reproduce here would a good > step forward. > > > > Would you describe the traffic contained in the pcap? > > Is it a single packet, or something that should hit a single DPCLS > > wildcarded > flow? > > > describe in comment above. Thanks - the details of the pcap should be enough for me to debug from here. <snip> > > > > > 5) pmd-stat-show > > > > > root@instance-3:~/ovs# ovs-appctl dpif-netdev/pmd-stats-show > > > > > pmd thread numa_id 0 core_id 0: > > > > > packets received: 19838528 > > > > > packet recirculations: 0 > > > > > avg. datapath passes per packet: 1.00 > > > > > emc hits: 0 > > > > > smc hits: 0 > > > > > megaflow hits: 0 > > > > > avg. subtable lookups per megaflow hit: 0.00 (---> this doesn't > > > > > look right ....) > > > > > miss with success upcall: 78 > > > > > miss with failed upcall: 19838418 > > > > > avg. packets per output batch: 2.00 > > > > > idle cycles: 0 (0.00%) > > > > > processing cycles: 103431787838 (100.00%) > > > > > avg cycles per packet: 5213.68 (103431787838/19838528) > > > > > avg processing cycles per packet: 5213.68 (103431787838/19838528) > > > > > > > > Would you try the pmd-stats-show command before setting the AVX512 > > > lookup? > Yes. > before setting avx512: > root@instance-3:~/ovs# ovs-appctl dpif-netdev/pmd-stats-show > pmd thread numa_id 0 core_id 0: > packets received: 70630720 > packet recirculations: 0 > avg. datapath passes per packet: 1.00 > emc hits: 64206054 > smc hits: 0 > megaflow hits: 6424309 > avg. subtable lookups per megaflow hit: 1.00 > miss with success upcall: 1 > miss with failed upcall: 324 > avg. packets per output batch: 0.00 > idle cycles: 1668002 (0.01%) > processing cycles: 17710219822 (99.99%) > avg cycles per packet: 250.77 (17711887824/70630720) > avg processing cycles per packet: 250.74 (17710219822/70630720) Thanks - this looks good - as expected. > > > > If the issue is still present it would indicate its not related to the > > > > exact > lookup > > > > implementation. > > > > > > Before setting AVX512 > > > ### Scalar Lookup > > > pmd thread numa_id 0 core_id 0: > > > packets received: 77470176 > > > packet recirculations: 0 > > > avg. datapath passes per packet: 1.00 > > > emc hits: 70423947 > > > smc hits: 0 > > > megaflow hits: 7045897 > > > avg. subtable lookups per megaflow hit: 1.00 > > > miss with success upcall: 1 > > > miss with failed upcall: 331 > > > avg. packets per output batch: 0.00 > > > idle cycles: 0 (0.00%) > > > processing cycles: 19596627706 (100.00%) > > > avg cycles per packet: 252.96 (19596627706/77470176) > > > avg processing cycles per packet: 252.96 (19596627706/77470176) > > > > > > ### AVX512 Lookup (restart ovs-vswitchd with additional command > > > "dpif-netdev/subtable-lookup-set avx512_gather 5" > > > pmd thread numa_id 0 core_id 0: > > > packets received: 1178784 > > > packet recirculations: 0 > > > avg. datapath passes per packet: 1.00 > > > emc hits: 0 > > > smc hits: 0 > > > megaflow hits: 0 > > > avg. subtable lookups per megaflow hit: 0.00 > > > miss with success upcall: 13 > > > miss with failed upcall: 1178739 ---> this looks not right > > > avg. packets per output batch: 0.00 > > > idle cycles: 0 (0.00%) > > > processing cycles: 5408870500 (100.00%) > > > avg cycles per packet: 4588.52 (5408870500/1178784) > > > avg processing cycles per packet: 4588.52 (5408870500/1178784) > > > > The statistics seem accurate (but indeed the upcall count is unexpected and > too high). > > This aligns with a ukey-install fail as noted in the logs above. > > > > This seems to indicate that with the AVX512 lookup the ukey install fails. > > I'd like to reproduce to investigate - above questions about traffic/rules > > is hopefully enough to identify. > > Why ukey is related here? Does you avx512 patch make any change to ukey? No AVX512 doesn't make any ukey changes - but issues in the hashing of the miniflow data blocks cause ukeys to be installed in different locations than where they are looked up - hence "ukey install fail" == "issue in miniflow iteration" in this context. > > There is an alternative - set the "autovalidator" DPCLS implementation to > > the highest priority, and it should ovs_assert() if the scalar/AVX512 > implementations > > mismatch. Then a dump of the OVS miniflow will give what's needed to verify > root cause. > > > that's a cool feature. > When setting > ovs-appctl dpif-netdev/subtable-lookup-set autovalidator 100 > log shows > 2020-05-21T22:28:55.964Z|77007|dpif_lookup_autovalidator(pmd- > c00/id:9)|ERR|matches_good > 7 != matches_test 0 for func avx512_gather Brilliant - this is exactly why the autovalidator is there. It has correctly flagged an issue here - I've reproduced using a pcap and your commands above. I will investigate a fix and include in the v3. Thanks for the details - will keep you all posted on progress. -Harry _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
