On Thu, Jun 8, 2023 at 3:32 AM Dumitru Ceara <[email protected]> wrote: > > On 6/8/23 07:57, Han Zhou wrote: > > On Wed, Jun 7, 2023 at 2:05 PM Numan Siddique <[email protected]> wrote: > >> > >> On Fri, Jun 2, 2023 at 12:12 AM Han Zhou <[email protected]> wrote: > >>> > >>> Maintain a set of lsp names that are referenced in the column > >>> "ip_port_mappings" of load_balancer records (used for health check > >>> purpose), which will be used for lsp incremental processing. > >>> > >>> Signed-off-by: Han Zhou <[email protected]> > >>> Reviewed-by: Ales Musil <[email protected]> > >> > >> Thanks for adding the lsp I-P in northd. > >> > >> LGTM. > >> > >> For this patch and for the rest of the patches (P7-P14) > >> > >> Acked-by: Numan Siddique <[email protected]> > >> > > Thanks Numan. I applied all P6-P14 to main. > > > > Han > > > > Hi Han, > > Sorry, I still didn't get a chance to review this series thoroughly, > I'll do that in-tree. However, I see some failures on one of my test > machines when running tests with latest main (this series applied). > > 1144: LSP incremental processing -- ovn-northd -- parallelization=yes FAILED (ovn-northd.at:9487) > 746: propagate Port_Binding.up to NB and OVS -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes FAILED (ovs-macros.at:258) > 748: propagate Port_Binding.up to NB and OVS -- ovn-northd -- parallelization=no -- ovn_monitor_all=yes FAILED (ovs-macros.at:258) > 747: propagate Port_Binding.up to NB and OVS -- ovn-northd -- parallelization=yes -- ovn_monitor_all=no FAILED (ovs-macros.at:258) > 749: propagate Port_Binding.up to NB and OVS -- ovn-northd -- parallelization=no -- ovn_monitor_all=no FAILED (ovs-macros.at:258) > > I didn't look into details yet but here's the testsuite log for > 746 and 1144 in case you want to check it out already. Bisecting > this it seems to point to: > > b41d74fcb382 ("northd: Ignore "up" column change if ignore_lsp_down is true.") > > Note: these tests pass fine on my laptop and in GitHub CI.. > > Regards, > Dumitru
Thanks Dumitru for the pointers. I submitted patches for the "Port_Binding.up" problem: https://patchwork.ozlabs.org/project/ovn/list/?series=359066 And for the unstable "LSP incremental processing" test: https://patchwork.ozlabs.org/project/ovn/patch/[email protected]/ Regards, Han > > 746. ovn.at:30779: testing propagate Port_Binding.up to NB and OVS -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes ... > creating ovn-sb database > ovsdb-server -vjsonrpc --remote=punix:/root/ovn/tests/testsuite.dir/0746/ovn-sb/ovn-sb.sock --remote=db:OVN_Southbound,SB_Global,connections --private-key=/root/ovn/tests/testpki-test-privkey.pem --certificate=/root/ovn/tests/testpki-test-cert.pem --ca-cert=/root/ovn/tests/testpki-cacert.pem /root/ovn/tests/testsuite.dir/0746/ovn-sb/ovn-sb.db -vconsole:off --detach --no-chdir --pidfile --log-file > creating ovn-nb database > ovsdb-server -vjsonrpc --remote=punix:/root/ovn/tests/testsuite.dir/0746/ovn-nb/ovn-nb.sock /root/ovn/tests/testsuite.dir/0746/ovn-nb/ovn-nb.db -vconsole:off --detach --no-chdir --pidfile --log-file > starting northd > ovn-northd --n-threads=4 -vjsonrpc --ovnnb-db=unix:/root/ovn/tests/testsuite.dir/0746/ovn-nb/ovn-nb.sock --ovnsb-db=unix:/root/ovn/tests/testsuite.dir/0746/ovn-sb/ovn-sb.sock -vconsole:off --detach --no-chdir --pidfile --log-file > 2023-06-08T09:50:57Z|00001|ovn_northd|INFO|Using 4 threads > starting northd-backup > ovn-northd --n-threads=4 -vjsonrpc --ovnnb-db=unix:/root/ovn/tests/testsuite.dir/0746/ovn-nb/ovn-nb.sock --ovnsb-db=unix:/root/ovn/tests/testsuite.dir/0746/ovn-sb/ovn-sb.sock -vconsole:off --detach --no-chdir --pidfile --log-file > 2023-06-08T09:50:57Z|00001|ovn_northd|INFO|Using 4 threads > 0ab11919-51d7-4258-b00b-865e8c497a00 > ovn-macros.at:234: waiting until TCP_PORT=`sed -n 's/.*0:.*: listening on port \([0-9]*\)$/\1/p' "$d/ovn-sb/ovsdb-server.log"` && test X != X"$TCP_PORT"... > ovn-macros.at:234: wait succeeded immediately > adding simulator 'main' > ovsdb-server --remote=punix:/root/ovn/tests/testsuite.dir/0746/main/db.sock -vconsole:off --detach --no-chdir --pidfile --log-file > ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif -vunixctl -vconsole:off --detach --no-chdir --pidfile --log-file > adding simulator 'hv1' > ovsdb-server --remote=punix:/root/ovn/tests/testsuite.dir/0746/hv1/db.sock -vconsole:off --detach --no-chdir --pidfile --log-file > ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif -vunixctl -vconsole:off --detach --no-chdir --pidfile --log-file > ovn-controller --enable-dummy-vif-plug -vconsole:off --detach --no-chdir --pidfile --log-file > ovn-nbctl ls-add ls > ./ovn-macros.at:419: "$@" > ## ----------------------------- ## > ## add OVS port for existing LSP ## > ## ----------------------------- ## > ovn-nbctl lsp-add ls lsp1 > ./ovn-macros.at:419: "$@" > ovn-nbctl --wait=hv sync > ./ovn-macros.at:419: "$@" > > Checking values in sb Port_Binding with logical_port=lsp1 against false... found false > ovs-vsctl add-port br-int lsp1 -- set Interface lsp1 external-ids:iface-id=lsp1 > ./ovn-macros.at:419: "$@" > > Waiting until up in sb Port_Binding with logical_port=lsp1 is true... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait succeeded immediately > > Waiting until up in nb Logical_Switch_Port with name=lsp1 is true... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait succeeded immediately > ovn.at:30779: waiting until test `ovs-vsctl get Interface lsp1 external_ids:ovn-installed` = '"true"'... > ovn.at:30779: wait succeeded immediately > ## ----------------------------- ## > ## add LSP for existing OVS port ## > ## ----------------------------- ## > ovs-vsctl add-port br-int lsp2 -- set Interface lsp2 external-ids:iface-id=lsp2 > ./ovn-macros.at:419: "$@" > ovn-nbctl lsp-add ls lsp2 > ./ovn-macros.at:419: "$@" > ovn-nbctl --wait=hv sync > ./ovn-macros.at:419: "$@" > > Checking values in sb Port_Binding with logical_port=lsp2 against true... found true > > Waiting until up in nb Logical_Switch_Port with name=lsp2 is true... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait succeeded immediately > ovn.at:30779: waiting until test `ovs-vsctl get Interface lsp2 external_ids:ovn-installed` = '"true"'... > ovn.at:30779: wait succeeded immediately > ## -------------------------------------------------------------- ## > ## ovn-controller should not reset Port_Binding.up without northd ## > ## -------------------------------------------------------------- ## > ovn-sbctl clear Port_Binding lsp1 up > ./ovn-macros.at:419: "$@" > ovn-sbctl clear Port_Binding lsp1 chassis > ./ovn-macros.at:419: "$@" > > Waiting until chassis in sb Port_Binding with logical_port=lsp1 is 3e5b1360-b992-48c3-874c-1e3203a2e6d3... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait succeeded immediately > > Checking values in sb Port_Binding with logical_port=lsp1 against ... found > > Waiting until up in sb Port_Binding with logical_port=lsp1 is true... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait succeeded immediately > > Waiting until up in nb Logical_Switch_Port with name=lsp1 is true... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait succeeded immediately > ## ------------------------------------------------------- ## > ## ovn-controller should reset Port_Binding.up - from NULL ## > ## ------------------------------------------------------- ## > ovn-sbctl clear Port_Binding lsp1 up > ./ovn-macros.at:419: "$@" > ovn-nbctl --wait=sb sync > ./ovn-macros.at:419: "$@" > > Waiting until up in nb Logical_Switch_Port with name=lsp1 is false... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait succeeded immediately > > Waiting until up in sb Port_Binding with logical_port=lsp1 is true... > ovn-macros.at:524: waiting until > found=$(ovn-${db}ctl --bare --columns $column find $table $a $b $c $d $e) > found=$(for d in $found; do echo $d; done | sort) > test "$expected" = "$found" > ... > ovn-macros.at:524: wait failed after 30 seconds > up in sb table Port_Binding has value , from the following rows: > _uuid : 54ce6370-38a3-452d-b2cb-7cb9bbb90e67 > additional_chassis : [] > additional_encap : [] > chassis : 3e5b1360-b992-48c3-874c-1e3203a2e6d3 > datapath : 0d18aa35-a297-4b3c-869d-1f71de7378cd > encap : [] > external_ids : {} > gateway_chassis : [] > ha_chassis_group : [] > logical_port : lsp2 > mac : [] > mirror_rules : [] > nat_addresses : [] > options : {} > parent_port : [] > port_security : [] > requested_additional_chassis: [] > requested_chassis : [] > tag : [] > tunnel_key : 2 > type : "" > up : true > virtual_parent : [] > > _uuid : b778b79c-1d82-478b-b93d-f0567e99ac41 > additional_chassis : [] > additional_encap : [] > chassis : 3e5b1360-b992-48c3-874c-1e3203a2e6d3 > datapath : 0d18aa35-a297-4b3c-869d-1f71de7378cd > encap : [] > external_ids : {} > gateway_chassis : [] > ha_chassis_group : [] > logical_port : lsp1 > mac : [] > mirror_rules : [] > nat_addresses : [] > options : {} > parent_port : [] > port_security : [] > requested_additional_chassis: [] > requested_chassis : [] > tag : [] > tunnel_key : 1 > type : "" > up : [] > virtual_parent : [] > ./ovs-macros.at:258: hard failure > 746. ovn.at:30779: 746. propagate Port_Binding.up to NB and OVS -- ovn-northd -- parallelization=yes -- ovn_monitor_all=yes (ovn.at:30779): FAILED (ovs-macros.at:258) > > --- > > 1144. ovn-northd.at:9487: testing LSP incremental processing -- ovn-northd -- parallelization=yes ... > creating ovn-sb database > ovsdb-server -vjsonrpc --remote=punix:/root/ovn/tests/testsuite.dir/1144/ovn-sb/ovn-sb.sock --remote=db:OVN_Southbound,SB_Global,connections --private-key=/root/ovn/tests/testpki-test-privkey.pem --certificate=/root/ovn/tests/testpki-test-cert.pem --ca-cert=/root/ovn/tests/testpki-cacert.pem /root/ovn/tests/testsuite.dir/1144/ovn-sb/ovn-sb.db -vconsole:off --detach --no-chdir --pidfile --log-file > creating ovn-nb database > ovsdb-server -vjsonrpc --remote=punix:/root/ovn/tests/testsuite.dir/1144/ovn-nb/ovn-nb.sock /root/ovn/tests/testsuite.dir/1144/ovn-nb/ovn-nb.db -vconsole:off --detach --no-chdir --pidfile --log-file > starting northd > ovn-northd --n-threads=4 -vjsonrpc --ovnnb-db=unix:/root/ovn/tests/testsuite.dir/1144/ovn-nb/ovn-nb.sock --ovnsb-db=unix:/root/ovn/tests/testsuite.dir/1144/ovn-sb/ovn-sb.sock -vconsole:off --detach --no-chdir --pidfile --log-file > 2023-06-08T09:56:03Z|00001|ovn_northd|INFO|Using 4 threads > starting northd-backup > ovn-northd --n-threads=4 -vjsonrpc --ovnnb-db=unix:/root/ovn/tests/testsuite.dir/1144/ovn-nb/ovn-nb.sock --ovnsb-db=unix:/root/ovn/tests/testsuite.dir/1144/ovn-sb/ovn-sb.sock -vconsole:off --detach --no-chdir --pidfile --log-file > 2023-06-08T09:56:03Z|00001|ovn_northd|INFO|Using 4 threads > b07d4382-d342-47b7-81b8-46fb188dc614 > ovn-macros.at:234: waiting until TCP_PORT=`sed -n 's/.*0:.*: listening on port \([0-9]*\)$/\1/p' "$d/ovn-sb/ovsdb-server.log"` && test X != X"$TCP_PORT"... > ovn-macros.at:234: wait succeeded immediately > adding simulator 'main' > ovsdb-server --remote=punix:/root/ovn/tests/testsuite.dir/1144/main/db.sock -vconsole:off --detach --no-chdir --pidfile --log-file > ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif -vunixctl -vconsole:off --detach --no-chdir --pidfile --log-file > adding simulator 'hv1' > ovsdb-server --remote=punix:/root/ovn/tests/testsuite.dir/1144/hv1/db.sock -vconsole:off --detach --no-chdir --pidfile --log-file > ovs-vswitchd --enable-dummy=system -vvconn -vofproto_dpif -vunixctl -vconsole:off --detach --no-chdir --pidfile --log-file > ovn-controller --enable-dummy-vif-plug -vconsole:off --detach --no-chdir --pidfile --log-file > ovn-nbctl --wait=hv ls-add ls0 > ./ovn-macros.at:419: "$@" > as northd ovn-appctl -t ovn-northd inc-engine/clear-stats > ./ovn-macros.at:419: "$@" > ovn-nbctl --wait=hv lsp-add ls0 lsp0-0 -- lsp-set-addresses lsp0-0 unknown > ./ovn-macros.at:419: "$@" > ovn-northd.at:9487: waiting until test `as northd ovn-appctl -t ovn-northd inc-engine/show-stats northd recompute` = 5... > ovn-northd.at:9487: wait succeeded immediately > ovn-northd.at:9487: waiting until test `as northd ovn-appctl -t ovn-northd inc-engine/show-stats lflow recompute` = 5... > ovn-northd.at:9487: wait succeeded immediately > as northd ovn-appctl -t ovn-northd inc-engine/clear-stats > ./ovn-macros.at:419: "$@" > ovn-nbctl --wait=hv lsp-add ls0 lsp0-1 -- lsp-set-addresses lsp0-1 aa:aa:aa:00:00:01 192.168.0.11 > ./ovn-macros.at:419: "$@" > ./ovn-northd.at:9487: as northd ovn-appctl -t ovn-northd inc-engine/show-stats northd recompute > ./ovn-northd.at:9487: as northd ovn-appctl -t ovn-northd inc-engine/show-stats lflow recompute > as northd ovn-appctl -t ovn-northd inc-engine/clear-stats > ./ovn-macros.at:419: "$@" > ovn-nbctl --wait=hv lsp-add ls0 lsp0-2 -- lsp-set-addresses lsp0-2 aa:aa:aa:00:00:02 192.168.0.12 > ./ovn-macros.at:419: "$@" > ./ovn-northd.at:9487: as northd ovn-appctl -t ovn-northd inc-engine/show-stats northd recompute > --- - 2023-06-08 05:56:06.104378452 -0400 > +++ /root/ovn/tests/testsuite.dir/at-groups/1144/stdout 2023-06-08 05:56:06.100714181 -0400 > @@ -1,2 +1,2 @@ > -0 > +1 > > 1144. ovn-northd.at:9487: 1144. LSP incremental processing -- ovn-northd -- parallelization=yes (ovn-northd.at:9487): FAILED (ovn-northd.at:9487) > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
