On 3/29/23 16:02, [email protected] wrote: > From: Numan Siddique <[email protected]> > > This patch adds a simple system test using ovn-fake-multinode > setup. The tests can be run as - 'make check-multinode' > > Before running these tests, user should deploy fake_multinode setup > by running 'ovn_cluster.sh start'. > > This test suite is also triggered for the newly added fake multinode CI > job. > > The fake multinode system tests suite can be enhanced further for new > features and to cover multi node scenarios. > > Signed-off-by: Numan Siddique <[email protected]> > ---
I have a few minor comments, feel free to add my ack when you apply the patch if you address them: Acked-by: Dumitru Ceara <[email protected]> Thanks, Dumitru > .../workflows/ovn-fake-multinode-tests.yml | 70 ++++++- > tests/automake.mk | 27 ++- > tests/multinode-macros.at | 190 ++++++++++++++++++ > tests/multinode-testsuite.at | 27 +++ > tests/multinode.at | 74 +++++++ > 5 files changed, 384 insertions(+), 4 deletions(-) > create mode 100644 tests/multinode-macros.at > create mode 100644 tests/multinode-testsuite.at > create mode 100644 tests/multinode.at > > diff --git a/.github/workflows/ovn-fake-multinode-tests.yml > b/.github/workflows/ovn-fake-multinode-tests.yml > index fa768f235c..ffc84f9d3f 100644 > --- a/.github/workflows/ovn-fake-multinode-tests.yml > +++ b/.github/workflows/ovn-fake-multinode-tests.yml > @@ -88,6 +88,13 @@ jobs: > CENTRAL_IMAGE: ${{ matrix.cfg.central_image }} > # Disable SSL for now. Revisit this if required. > ENABLE_SSL: no > + CC: gcc > + OPTS: "--disable-ssl" > + dependencies: | > + automake libtool gcc bc libjemalloc2 libjemalloc-dev \ > + libssl-dev llvm-dev libelf-dev libnuma-dev libpcap-dev \ > + selinux-policy-dev ncat python3-scapy isc-dhcp-server \ > + podman openvswitch-switch libunbound-dev libunwind-dev > # https://github.com/actions/runner-images/issues/6282 > XDG_RUNTIME_DIR: '' > > @@ -101,6 +108,8 @@ jobs: > - { central_image: "ovn/ovn-multi-node:22.03" } > > steps: > + - name: install required dependencies > + run: sudo apt install -y ${{ env.dependencies }} > > - name: Free up disk space > run: sudo eatmydata apt-get remove --auto-remove -y aspnetcore-* > dotnet-* libmono-* mono-* msbuild php-* php7* ghc-* zulu-* > @@ -125,10 +134,8 @@ jobs: > path: 'ovn-fake-multinode' > ref: 'v0.1' > > - - name: Install dependencies > + - name: Start openvswitch > run: | > - sudo apt update > - sudo apt-get install -y podman openvswitch-switch > sudo systemctl start openvswitch-switch > sudo ovs-vsctl show > > @@ -145,6 +152,63 @@ jobs: > sudo ./.ci/test_basic.sh > working-directory: ovn-fake-multinode > > + - name: update PATH > + run: | > + echo "$HOME/bin" >> $GITHUB_PATH > + echo "$HOME/.local/bin" >> $GITHUB_PATH > + > + - name: set up python > + uses: actions/setup-python@v4 > + with: > + python-version: '3.x' > + > + - name: Check out ovn > + uses: actions/checkout@v3 > + with: > + path: 'ovn' > + submodules: recursive > + > + - name: Build OVN and trigger fake-multinode system tests > + run: | > + set -x > + pwd > + cd ovn > + ./.ci/linux-prepare.sh > + ./.ci/linux-build.sh > + sudo make check-multinode || : > + sudo podman exec -it ovn-central ovn-nbctl show || : > + sudo podman exec -it ovn-central ovn-sbctl show || : > + sudo podman exec -it ovn-chassis-1 ovs-vsctl show || : > + sudo podman exec -it ovn-chassis-1 ip netns || : > + sudo podman exec -it ovn-chassis-1 cat > /var/log/ovn/ovn-controller.log || : > + sudo cat tests/multinode-testsuite.dir/1/multinode-testsuite.log || : > + > + - name: copy logs on failure > + if: failure() || cancelled() > + run: | > + # upload-artifact@v3 throws exceptions if it tries to upload socket > + # files and we could have some socket files in testsuite.dir. > + # Also, upload-artifact@v3 doesn't work well enough with wildcards. > + # So, we're just archiving everything here to avoid any issues. > + pwd > + ls -l These two lines can be removed I guess. > + mkdir logs > + ls -l ovn/ > + ls -l ovn/tests/ Same here. > + > + cp ovn/config.log ./logs/ > + # multinode tests are run as root, need to adjust permissions. > + sudo chmod -R +r ovn/tests/multinode-testsuite.dir.* || true > + cp -r ovn/tests/multinode-testsuite.dir.* ./logs/ || true > + tar -czvf logs.tgz logs/ > + > + - name: upload logs on failure > + if: failure() || cancelled() > + uses: actions/upload-artifact@v3 > + with: > + name: logs-linux-${{ join(matrix.cfg.*, '-') }} > + path: logs.tgz > + > - name: Stop cluster > run: | > sudo -E ./ovn_cluster.sh stop > diff --git a/tests/automake.mk b/tests/automake.mk > index d9f2777f34..fd5ee14afd 100644 > --- a/tests/automake.mk > +++ b/tests/automake.mk > @@ -5,10 +5,12 @@ EXTRA_DIST += \ > $(SYSTEM_KMOD_TESTSUITE_AT) \ > $(SYSTEM_USERSPACE_TESTSUITE_AT) \ > $(PERF_TESTSUITE_AT) \ > + $(MULTINODE_TESTSUITE_AT) \ > $(TESTSUITE) \ > $(SYSTEM_KMOD_TESTSUITE) \ > $(SYSTEM_USERSPACE_TESTSUITE) \ > $(PERF_TESTSUITE) \ > + $(MULTINODE_TESTSUITE) \ > tests/atlocal.in \ > $(srcdir)/package.m4 \ > $(srcdir)/tests/testsuite \ > @@ -61,6 +63,11 @@ PERF_TESTSUITE_AT = \ > tests/perf-testsuite.at \ > tests/perf-northd.at > > +MULTINODE_TESTSUITE_AT = \ > + tests/multinode-testsuite.at \ > + tests/multinode-macros.at \ > + tests/multinode.at > + > check_SCRIPTS += tests/atlocal > > TESTSUITE = $(srcdir)/tests/testsuite > @@ -72,7 +79,9 @@ PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite > PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir > PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results > DISTCLEANFILES += tests/atconfig tests/atlocal > - > +MULTINODE_TESTSUITE = $(srcdir)/tests/multinode-testsuite > +MULTINODE_TESTSUITE_DIR = $(abs_top_builddir)/tests/multinode-testsuite.dir > +MULTINODE_TESTSUITE_RESULTS = $(MULTINODE_TESTSUITE_DIR)/results > AUTOTEST_PATH = > $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic > > export ovs_srcdir > @@ -197,6 +206,18 @@ check-perf: all > @echo > @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" > > +check-multinode: all > + @mkdir -p $(MULTINODE_TESTSUITE_DIR) > + @echo > $(MULTINODE_TESTSUITE_RESULTS) > + set $(SHELL) '$(MULTINODE_TESTSUITE)' -C tests > AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ > + "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && "$$@" > --recheck) > + @echo > + @echo '## -------------------- ##' > + @echo '## Multinode test Results ##' > + @echo '## -------------------- ##' > + @cat $(MULTINODE_TESTSUITE_RESULTS) > + @echo > + @echo "Results can be found in $(MULTINODE_TESTSUITE_RESULTS)" > > AUTOTEST = $(AUTOM4TE) --language=autotest > > @@ -223,6 +244,10 @@ $(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) > $(COMMON_MACROS_AT) > $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o [email protected] [email protected] > $(AM_V_at)mv [email protected] $@ > > +$(MULTINODE_TESTSUITE): package.m4 $(MULTINODE_TESTSUITE_AT) > $(COMMON_MACROS_AT) > + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o [email protected] [email protected] > + $(AM_V_at)mv [email protected] $@ > + > # The `:;' works around a Bash 3.2 bug when the output is not writeable. > $(srcdir)/package.m4: $(top_srcdir)/configure.ac > $(AM_V_GEN):;{ \ > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > new file mode 100644 > index 0000000000..1525442804 > --- /dev/null > +++ b/tests/multinode-macros.at > @@ -0,0 +1,190 @@ > +# > +# > + > +# M_NS_EXEC([fake_node], [namespace], [command]) > +# > +# Execute 'command' in 'namespace' > +m4_define([M_NS_EXEC], > + [podman exec $1 ip netns exec $2 $3]) > + > +# M_NS_CHECK_EXEC([fake_node], [namespace], [command], other_params...) > +# > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s > namespace'. > +# 'other_params' as passed as they are to AT_CHECK. > +m4_define([M_NS_CHECK_EXEC], > + [ AT_CHECK([M_NS_EXEC([$1], [$2], [$3])], > m4_shift(m4_shift(m4_shift($@)))) ] > +) > + > +OVS_START_SHELL_HELPERS > + > +m_as() { > + c=$1 > + shift > + podman exec $c "$@" > +} > + > +m_central_as () { > + podman exec ovn-central "$@" > +} > + > +check_fake_multinode_setup() { > + check m_as ovn-central ovn-nbctl --wait=sb sync > + AT_CHECK([m_as ovn-chassis-1 ovn-appctl -t ovn-controller version], [0], > [ignore]) > + AT_CHECK([m_as ovn-chassis-2 ovn-appctl -t ovn-controller version], [0], > [ignore]) > + AT_CHECK([m_as ovn-gw-1 ovn-appctl -t ovn-controller version], [0], > [ignore]) > + AT_CHECK([m_as ovn-gw-1 ovn-appctl -t ovn-controller version], [0], > [ignore]) > +} > + > +cleanup_multinode_resources() { > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > + check m_as ovn-central ovn-nbctl --wait=sb sync > + for c in ovn-chassis-1 ovn-chassis-2 ovn-gw-1 > + do > + m_as $c ovs-vsctl del-br br-int > + m_as $c ip --all netns delete > + done > +} > + > +multinode_nbctl () { > + m_as ovn-central ovn-nbctl "$@" > +} > + > +# count_rows TABLE [CONDITION...] Nit: m_count_rows > +# > +# Prints the number of rows in TABLE (that satisfy CONDITION). > +# Uses the southbound db by default; set DB=nb for the northbound database. > +m_count_rows() { > + local db=$(parse_db $1) table=$(parse_table $1); shift > + m_central_as ovn-${db}ctl --format=table --no-headings find $table "$@" > | wc -l > +} > + > +# check_row_count [DATABASE:]TABLE COUNT [CONDITION...] Nit: m_check_row_count > +# > +# Checks that TABLE contains COUNT rows (that satisfy CONDITION). > +# The default DATABASE is "sb". > +m_check_row_count() { > + local db=$(parse_db $1) table=$(parse_table $1); shift > + local count=$1; shift > + local found=$(m_count_rows $c $db:$table "$@") > + echo > + echo "Checking for $count rows in $db $table${1+ with $*}... found > $found" > + if test "$count" != "$found"; then > + m_central_as ovn-${db}ctl list $table > + AT_FAIL_IF([:]) > + fi > +} > + > +# wait_row_count [DATABASE:]TABLE COUNT [CONDITION...] Nit: m_wait_row_count > +# > +# Waits until TABLE contains COUNT rows (that satisfy CONDITION). > +# The default DATABASE is "sb". > +m_wait_row_count() { > + local db=$(parse_db $1) table=$(parse_table $1); shift > + local count=$1; shift > + local a=$1 b=$2 c=$3 d=$4 e=$5 > + echo "Waiting until $count rows in $db $table${1+ with $*}..." > + OVS_WAIT_UNTIL([test $count = $(m_count_rows $db:$table $a $b $c $d > $e)],[ > + echo "$db table $table has the following rows. $(m_count_rows > $db:$table $a $b $c $d $e) rows match instead of expected $count:" > + m_central_as ovn-${db}ctl list $table]) > +} > + > +# multinode_wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]] Nit: m_wait_column > +# > +# Wait until all of the values of COLUMN in the rows of TABLE (that > +# satisfy CONDITION) equal EXPECTED (ignoring order). > +# > +# The default DATABASE is "sb". > +# > +# COLUMN defaults to _uuid if unspecified. > +m_wait_column() { > + local expected=$(for d in $1; do echo $d; done | sort) > + local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; > shift; shift; shift > + local a=$1 b=$2 c=$3 d=$4 e=$5 > + > + echo > + echo "Waiting until $column in $db $table${1+ with $*} is $expected..." > + OVS_WAIT_UNTIL([ > + found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table > $a $b $c $d $e) > + found=$(for d in $found; do echo $d; done | sort) > + test "$expected" = "$found" > + ], [ > + echo "$column in $db table $table has value $found, from the following > rows:" > + m_central_as ovn-${db}ctl list $table]) > +} > + > +# fetch_column [DATABASE:]TABLE COLUMN [CONDITION...] Nit: m_fetch_column > +# > +# Fetches and prints all the values of COLUMN in the rows of TABLE > +# (that satisfy CONDITION), sorting the results lexicographically. > +# The default DATABASE is "sb". > +m_fetch_column() { > + local db=$(parse_db $1) table=$(parse_table $1) column=${2-_uuid}; > shift; shift > + # Using "echo" removes spaces and newlines. > + echo $(m_central_as ovn-${db}ctl --bare --columns $column find $table > "$@" | sort) > +} > + > +# check_column EXPECTED [DATABASE:]TABLE COLUMN [CONDITION...] Nit: m_check_column > +# > +# Fetches all of the values of COLUMN in the rows of TABLE (that > +# satisfy CONDITION), and compares them against EXPECTED (ignoring > +# order). > +# > +# The default DATABASE is "sb". > +m_check_column() { > + local expected=$1 db=$(parse_db $2) table=$(parse_table $2) > column=${3-_uuid}; shift; shift; shift > + local found=$(m_central_as ovn-${db}ctl --bare --columns $column find > $table "$@") > + > + # Sort the expected and found values. > + local found=$(for d in $found; do echo $d; done | sort) > + local expected=$(for d in $expected; do echo $d; done | sort) > + > + echo > + echo "Checking values in $db $table${1+ with $*} against $expected... > found $found" > + if test "$found" != "$expected"; then > + m_central_as ovn-${db}ctl list $table > + AT_FAIL_IF([:]) > + fi > +} > + > +# wait_column EXPECTED [DATABASE:]TABLE [COLUMN [CONDITION...]] Nit: m_wait_column > +# > +# Wait until all of the values of COLUMN in the rows of TABLE (that > +# satisfy CONDITION) equal EXPECTED (ignoring order). > +# > +# The default DATABASE is "sb". > +# > +# COLUMN defaults to _uuid if unspecified. > +m_wait_column() { > + local expected=$(for d in $1; do echo $d; done | sort) > + local db=$(parse_db $2) table=$(parse_table $2) column=${3-_uuid}; > shift; shift; shift > + local a=$1 b=$2 c=$3 d=$4 e=$5 > + > + echo > + echo "Waiting until $column in $db $table${1+ with $*} is $expected..." > + OVS_WAIT_UNTIL([ > + found=$(m_central_as ovn-${db}ctl --bare --columns $column find $table > $a $b $c $d $e) > + found=$(for d in $found; do echo $d; done | sort) > + test "$expected" = "$found" > + ], [ > + echo "$column in $db table $table has value $found, from the following > rows:" > + m_central_as ovn-${db}ctl list $table]) > +} > + > +# wait_for_ports_up [PORT...] Nit: m_wait_for_ports_up > +# > +# With arguments, waits for specified Logical_Switch_Ports to come up. > +# Without arguments, waits for all "plain" and router > +# Logical_Switch_Ports to come up. > +m_wait_for_ports_up() { > + if test $# = 0; then > + m_wait_row_count nb:Logical_Switch_Port 0 up!=true type='""' > + m_wait_row_count nb:Logical_Switch_Port 0 up!=true type=router > + else > + for port; do > + m_wait_row_count nb:Logical_Switch_Port 1 up=true name=$port > + done > + fi > +} > + > +OVS_END_SHELL_HELPERS > diff --git a/tests/multinode-testsuite.at b/tests/multinode-testsuite.at > new file mode 100644 > index 0000000000..ea10b02764 > --- /dev/null > +++ b/tests/multinode-testsuite.at > @@ -0,0 +1,27 @@ > +AT_INIT > + > +AT_COPYRIGHT([Copyright (c) 2022 Red Hat, > + > +Licensed under the Apache License, Version 2.0 (the "License"); > +you may not use this file except in compliance with the License. > +You may obtain a copy of the License at: > + > + http://www.apache.org/licenses/LICENSE-2.0 > + > +Unless required by applicable law or agreed to in writing, software > +distributed under the License is distributed on an "AS IS" BASIS, > +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > +See the License for the specific language governing permissions and > +limitations under the License.]) > + > +m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS]) > +AT_ARG_OPTION([rebuild], [Do not use cached versions of databases]) > + > +m4_include([tests/ovs-macros.at]) > +m4_include([tests/ovsdb-macros.at]) > +m4_include([tests/ofproto-macros.at]) > +m4_include([tests/ovn-macros.at]) > +m4_include([tests/system-common-macros.at]) > +m4_include([tests/multinode-macros.at]) > + > +m4_include([tests/multinode.at]) > diff --git a/tests/multinode.at b/tests/multinode.at > new file mode 100644 > index 0000000000..2b199b4bce > --- /dev/null > +++ b/tests/multinode.at > @@ -0,0 +1,74 @@ > +AT_BANNER([ovn multinode system tests using ovn-fake-multinode]) > + > +AT_SETUP([ovn multinode basic test]) Do we need something like this here? AT_SKIP_IF([test $HAVE_PODMAN = no]) > + > +# Check that ovn-fake-multinode setup is up and running > +check_fake_multinode_setup > + > +# Delete the multinode NB and OVS resources before starting the test. > +cleanup_multinode_resources > + > +# Test East-West switching > +check multinode_nbctl ls-add sw0 > +check multinode_nbctl lsp-add sw0 sw0-port1 > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > 10.0.0.3 1000::3" > +check multinode_nbctl lsp-add sw0 sw0-port2 > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > 10.0.0.4 1000::4" > + > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 > 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 50:54:00:00:00:04 > 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > + > +m_wait_for_ports_up > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 > | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +# Add ACLs to drop all traffic > +check multinode_nbctl pg-add pg0 sw0-port1 sw0-port2 > +check multinode_nbctl acl-add pg0 to-lport 1001 "outport == @pg0 && ip4" drop > +check multinode_nbctl --wait=sb sync > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > 10.0.0.4], \ > +[1], [ignore]) > + > +# Add ACLs to allow icmp traffic > +check multinode_nbctl acl-add pg0 to-lport 1002 "outport == @pg0 && ip4 && > icmp" allow-related > +check multinode_nbctl --wait=sb sync > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 > | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > + > +# Create the second logical switch with one port > +check multinode_nbctl ls-add sw1 > +check multinode_nbctl lsp-add sw1 sw1-port1 > +check multinode_nbctl lsp-set-addresses sw1-port1 "40:54:00:00:00:03 > 20.0.0.3 2000::3" > + > +# Create a logical router and attach both logical switches > +check multinode_nbctl lr-add lr0 > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 > 1000::a/64 > +check multinode_nbctl lsp-add sw0 sw0-lr0 > +check multinode_nbctl lsp-set-type sw0-lr0 router > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > + > +check multinode_nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24 > 2000::a/64 > +check multinode_nbctl lsp-add sw1 sw1-lr0 > +check multinode_nbctl lsp-set-type sw1-lr0 router > +check multinode_nbctl lsp-set-addresses sw1-lr0 router > +check multinode_nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1 > + > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw1-port1 sw1p1 40:54:00:00:00:03 > 20.0.0.3 24 20.0.0.1 2000::4/64 1000::a > + > +m_wait_for_ports_up sw1-port1 > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 20.0.0.3 > | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +AT_CLEANUP _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
