On 12 Jul 2021, at 7:51, kumar Amber wrote:
> From: Kumar Amber <[email protected]> > > Tests: > 6: OVS-DPDK - MFEX Autovalidator > 7: OVS-DPDK - MFEX Autovalidator Fuzzy > > Added a new directory to store the PCAP file used > in the tests and a script to generate the fuzzy traffic > type pcap to be used in fuzzy unit test. > > Signed-off-by: Kumar Amber <[email protected]> > Acked-by: Flavio Leitner <[email protected]> > --- > v7: > - fix review comments(Eelco) > v5: > - fix review comments(Ian, Flavio, Eelco) > - remove sleep from first test and added minor 5 sec sleep to fuzzy > --- > --- > Documentation/topics/dpdk/bridge.rst | 55 +++++++++++++++++++++++++++ > tests/.gitignore | 1 + > tests/automake.mk | 5 +++ > tests/mfex_fuzzy.py | 31 +++++++++++++++ > tests/pcap/mfex_test.pcap | Bin 0 -> 416 bytes > tests/system-dpdk.at | 49 ++++++++++++++++++++++++ > 6 files changed, 141 insertions(+) > create mode 100755 tests/mfex_fuzzy.py > create mode 100644 tests/pcap/mfex_test.pcap > > diff --git a/Documentation/topics/dpdk/bridge.rst > b/Documentation/topics/dpdk/bridge.rst > index c31067c51..f33bd8203 100644 > --- a/Documentation/topics/dpdk/bridge.rst > +++ b/Documentation/topics/dpdk/bridge.rst > @@ -345,3 +345,58 @@ A compile time option is available in order to test it > with the OVS unit > test suite. Use the following configure option :: > > $ ./configure --enable-mfex-default-autovalidator > + > +Unit Test Miniflow Extract > +++++++++++++++++++++++++++ > + > +Unit test can also be used to test the workflow mentioned above by running > +the following test-case in tests/system-dpdk.at :: > + > + make check-dpdk TESTSUITEFLAGS='-k MFEX' > + OVS-DPDK - MFEX Autovalidator > + > +The unit test uses mulitple traffic type to test the correctness of the > +implementaions. > + > +Running Fuzzy test with Autovalidator > ++++++++++++++++++++++++++++++++++++++ > + > +Fuzzy tests can also be done on miniflow extract with the help of > +auto-validator and Scapy. The steps below describes the steps to > +reproduce the setup with IP being fuzzed to generate packets. > + > +Scapy is used to create fuzzy IP packets and save them into a PCAP :: > + > + pkt = fuzz(Ether()/IP()/TCP()) > + > +Set the miniflow extract to autovalidator using :: > + > + $ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator > + > +OVS is configured to receive the generated packets :: > + > + $ ovs-vsctl add-port br0 pcap0 -- \ > + set Interface pcap0 type=dpdk options:dpdk-devargs=net_pcap0 > + "rx_pcap=fuzzy.pcap" > + > +With this workflow, the autovalidator will ensure that all MFEX > +implementations are classifying each packet in exactly the same way. > +If an optimized MFEX implementation causes a different miniflow to be > +generated, the autovalidator has ovs_assert and logging statements that > +will inform about the issue. > + > +Unit Fuzzy test with Autovalidator > ++++++++++++++++++++++++++++++++++++++ > + > +The prerquiste before running the unit test is to run the script provided :: > + > + tests/mfex_fuzzy.py > + > +This script generates a pcap with mulitple type of fuzzed packets to be used > +in the below unit test-case. This is no longer true right, as it will be run automatically by the unit test. > + > +Unit test can also be used to test the workflow mentioned above by running > +the following test-case in tests/system-dpdk.at :: > + > + make check-dpdk TESTSUITEFLAGS='-k MFEX' > + OVS-DPDK - MFEX Autovalidator Fuzzy > diff --git a/tests/.gitignore b/tests/.gitignore > index 45b4f67b2..a3d927e5d 100644 > --- a/tests/.gitignore > +++ b/tests/.gitignore > @@ -11,6 +11,7 @@ > /ovsdb-cluster-testsuite > /ovsdb-cluster-testsuite.dir/ > /ovsdb-cluster-testsuite.log > +/pcap/ > /pki/ > /system-afxdp-testsuite > /system-afxdp-testsuite.dir/ > diff --git a/tests/automake.mk b/tests/automake.mk > index f45f8d76c..2bcf054b0 100644 > --- a/tests/automake.mk > +++ b/tests/automake.mk > @@ -143,6 +143,11 @@ $(srcdir)/tests/fuzz-regression-list.at: > tests/automake.mk > echo "TEST_FUZZ_REGRESSION([$$basename])"; \ > done > [email protected] && mv [email protected] $@ > > +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS) > +MFEX_AUTOVALIDATOR_TESTS = \ > + tests/pcap/mfex_test.pcap \ > + tests/mfex_fuzzy.py > + > OVSDB_CLUSTER_TESTSUITE_AT = \ > tests/ovsdb-cluster-testsuite.at \ > tests/ovsdb-execution.at \ > diff --git a/tests/mfex_fuzzy.py b/tests/mfex_fuzzy.py > new file mode 100755 > index 000000000..395158b0d > --- /dev/null > +++ b/tests/mfex_fuzzy.py I opened the below python file in emacs and got tons of flake warnings. Looking at the makefile, I noticed you forgot to add it to the flake list. Please make the following modification: diff --git a/tests/automake.mk b/tests/automake.mk index 2bcf054b0..0ae7529d0 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -530,7 +530,8 @@ CHECK_PYFILES = \ tests/test-unixctl.py \ tests/test-vlog.py \ tests/uuidfilt.py \ - tests/sendpkt.py + tests/sendpkt.py \ + tests/mfex_fuzzy.py EXTRA_DIST += $(CHECK_PYFILES) PYCOV_CLEAN_FILES += $(CHECK_PYFILES:.py=.py,cover) .coverage And fix the errors: tests/mfex_fuzzy.py:3:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:3:4: F403 'from scapy.all import *' used; unable to detect undefined names tests/mfex_fuzzy.py:5:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:9:11: F405 'PcapWriter' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:13:4: E114 indentation is not a multiple of four (comment) tests/mfex_fuzzy.py:13:80: E501 line too long (92 > 79 characters) tests/mfex_fuzzy.py:14:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:14:10: F405 'Ether' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:14:20: F405 'RandMAC' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:14:35: F405 'RandMAC' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:15:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:15:11: F405 'Dot1Q' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:16:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:16:11: F405 'IP' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:16:18: F405 'RandIP' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:16:32: F405 'RandIP' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:17:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:17:11: F405 'IPv6' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:17:20: F405 'RandIP6' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:17:35: F405 'RandIP6' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:18:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:18:10: F405 'UDP' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:18:20: F405 'RandShort' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:18:39: F405 'RandShort' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:19:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:19:10: F405 'TCP' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:19:20: F405 'RandShort' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:19:39: F405 'RandShort' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:21:4: E114 indentation is not a multiple of four (comment) tests/mfex_fuzzy.py:22:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:22:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:22:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:22:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:23:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:23:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:23:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:23:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:24:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:24:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:24:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:24:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:24:36: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:25:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:25:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:25:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:25:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:25:36: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:27:5: E116 unexpected indentation (comment) tests/mfex_fuzzy.py:28:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:28:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:28:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:28:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:29:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:29:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:29:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:29:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:30:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:30:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:30:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:30:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:30:36: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:31:4: E111 indentation is not a multiple of four tests/mfex_fuzzy.py:31:18: F405 'fuzz' may be undefined, or defined from star imports: scapy.all tests/mfex_fuzzy.py:31:26: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:31:31: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:31:36: E226 missing whitespace around arithmetic operator tests/mfex_fuzzy.py:31:42: W292 no newline at end of file > @@ -0,0 +1,31 @@ > +#!/usr/bin/python3 > +try: > + from scapy.all import * > +except ModuleNotFoundError as err: > + print(err + ": Scapy") > +import sys > + > +path = str(sys.argv[1]) + "/pcap/fuzzy.pcap" > +pktdump = PcapWriter(path, append=False, sync=True) > + > +for i in range(0, 2000): > + > + # Generate random protocol bases, use a fuzz() over the combined packet > for full fuzzing. > + eth = Ether(src=RandMAC(), dst=RandMAC()) > + vlan = Dot1Q() > + ipv4 = IP(src=RandIP(), dst=RandIP()) > + ipv6 = IPv6(src=RandIP6(), dst=RandIP6()) > + udp = UDP(dport=RandShort(), sport=RandShort()) > + tcp = TCP(dport=RandShort(), sport=RandShort()) > + > + # IPv4 packets with fuzzing > + pktdump.write(fuzz(eth/ipv4/udp)) > + pktdump.write(fuzz(eth/ipv4/tcp)) > + pktdump.write(fuzz(eth/vlan/ipv4/udp)) > + pktdump.write(fuzz(eth/vlan/ipv4/tcp)) > + > + # IPv6 packets with fuzzing > + pktdump.write(fuzz(eth/ipv6/udp)) > + pktdump.write(fuzz(eth/ipv6/tcp)) > + pktdump.write(fuzz(eth/vlan/ipv6/udp)) > + pktdump.write(fuzz(eth/vlan/ipv6/tcp)) > \ No newline at end of file > diff --git a/tests/pcap/mfex_test.pcap b/tests/pcap/mfex_test.pcap > new file mode 100644 > index > 0000000000000000000000000000000000000000..1aac67b8d643ecb016c758cba4cc32212a80f52a > GIT binary patch > literal 416 > zcmca|c+)~A1{MYw`2U}Qff2}Q<eHVR>K`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o| > z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$<F=a4i;9x8Q*<ZrSM6Ufz > xK>KST_NTIwYriok6N4Vm)gX-Q@<yO<!C`>c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW > > literal 0 > HcmV?d00001 > > diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at > index 802895488..55fd7baa3 100644 > --- a/tests/system-dpdk.at > +++ b/tests/system-dpdk.at > @@ -232,3 +232,52 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch > kernel module is probably > \@EAL: No free hugepages reported in hugepages-1048576kB@d"]) > AT_CLEANUP > dnl > -------------------------------------------------------------------------- > + > +dnl > -------------------------------------------------------------------------- > +dnl Add standard DPDK PHY port > +AT_SETUP([OVS-DPDK - MFEX Autovalidator]) What happened to this conversation? “”” > I think we should also skip these tests if we do not have a machine > that has AVX512. Just to make sure we do not generate an OK where we > are not even testing the AVX512 functions. > > Actually we should not what if someone wants to write a new mfex > version without AVX but just SIMD or some other way than we are > probably blocking the testing Good catch! I think we should run the test if other implementations are available, else skip (except for auto, scalar, and study as they are always available). “”” > +AT_KEYWORDS([dpdk]) > + > +OVS_DPDK_START() > + > +dnl Add userspace bridge and attach it to OVS > +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev]) > +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk > options:dpdk-devargs=net_pcap1,rx_pcap=$srcdir/pcap/mfex_test.pcap,infinite_rx=1], > [], [stdout], [stderr]) > +AT_CHECK([ovs-vsctl show], [], [stdout]) > + > + > +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], > [dnl > +Miniflow implementation set to autovalidator. > +]) > + > +OVS_WAIT_UNTIL([test `ovs-vsctl get interface p1 statistics | grep -oP > 'rx_packets=\s*\K\d+'` -ge 1000]) > + > +dnl Clean up > +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr]) > +AT_CLEANUP > +dnl > -------------------------------------------------------------------------- > + > +dnl > -------------------------------------------------------------------------- > +dnl Add standard DPDK PHY port > +AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy]) > +AT_KEYWORDS([dpdk]) > +AT_SKIP_IF([! pip3 list | grep scapy], [], []) > +AT_CHECK([$PYTHON3 $srcdir/mfex_fuzzy.py $srcdir], [], [stdout]) > +OVS_DPDK_START() > + > +dnl Add userspace bridge and attach it to OVS > +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev]) > +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk > options:dpdk-devargs=net_pcap1,rx_pcap=$srcdir/pcap/fuzzy.pcap,infinite_rx=1], > [], [stdout], [stderr]) > +AT_CHECK([ovs-vsctl show], [], [stdout]) > + > + > +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], > [dnl > +Miniflow implementation set to autovalidator. > +]) > + > +OVS_WAIT_UNTIL([test `ovs-vsctl get interface p1 statistics | grep -oP > 'rx_packets=\s*\K\d+'` -ge 100000]) > + > +dnl Clean up > +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr]) > +AT_CLEANUP > +dnl > -------------------------------------------------------------------------- > -- > 2.25.1 _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
