Re: [ovs-dev] ovn master test failures
It was great fun, I haven't had a nice satisfying bug hunt in a while. On Fri, May 14, 2021 at 03:39:22PM -0400, Ihar Hrachyshka wrote: > Thank you Ben, > > an impressive feat and an interesting write-up. :) > > Ihar > > On Fri, May 14, 2021 at 12:21 AM Ben Pfaff wrote: > > > > On Wed, May 12, 2021 at 09:21:32PM -0400, Ihar Hrachyshka wrote: > > > Hi Numan, > > > > > > both 3187b9fef124e038e474270a2728fe94bdca8eef (ovn-northd: introduce > > > new allow-stateless ACL verb) and > > > 127bf166ccf4a2509f670c48a00b0340039f20d2 (northd: Support flow > > > offloading for logical switches with no ACLs.) got merged in upstream > > > master, and this combination broke the following tests: > > > > > > 774: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > > > ovn-northd-ddlog -- dp-groups=yes FAILED (ovn-northd.at:2752) > > > 775: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > > > ovn-northd-ddlog FAILED (ovn-northd.at:2752) > > > > > > while the other scenarios are passing: > > > > > > 768: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > > ovn-northd -- dp-groups=yes ok > > > 769: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > > ovn-northd ok > > > 770: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > > ovn-northd-ddlog -- dp-groups=yes ok > > > 771: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > > ovn-northd-ddlog ok > > > 772: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > > > ovn-northd -- dp-groups=yes ok > > > 773: ovn -- ACL allow-stateless omit conntrack - Port_Group -- ovn-northd > > > ok > > > > This turned to out to be an all-afternoon-and-evening saga, but I got it > > fixed and documented: > > https://mail.openvswitch.org/pipermail/ovs-dev/2021-May/382993.html > > https://github.com/vmware/differential-datalog/pull/977 > > > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] ovn master test failures
Thank you Ben, an impressive feat and an interesting write-up. :) Ihar On Fri, May 14, 2021 at 12:21 AM Ben Pfaff wrote: > > On Wed, May 12, 2021 at 09:21:32PM -0400, Ihar Hrachyshka wrote: > > Hi Numan, > > > > both 3187b9fef124e038e474270a2728fe94bdca8eef (ovn-northd: introduce > > new allow-stateless ACL verb) and > > 127bf166ccf4a2509f670c48a00b0340039f20d2 (northd: Support flow > > offloading for logical switches with no ACLs.) got merged in upstream > > master, and this combination broke the following tests: > > > > 774: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > > ovn-northd-ddlog -- dp-groups=yes FAILED (ovn-northd.at:2752) > > 775: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > > ovn-northd-ddlog FAILED (ovn-northd.at:2752) > > > > while the other scenarios are passing: > > > > 768: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > ovn-northd -- dp-groups=yes ok > > 769: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > ovn-northd ok > > 770: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > ovn-northd-ddlog -- dp-groups=yes ok > > 771: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > > ovn-northd-ddlog ok > > 772: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > > ovn-northd -- dp-groups=yes ok > > 773: ovn -- ACL allow-stateless omit conntrack - Port_Group -- ovn-northd ok > > This turned to out to be an all-afternoon-and-evening saga, but I got it > fixed and documented: > https://mail.openvswitch.org/pipermail/ovs-dev/2021-May/382993.html > https://github.com/vmware/differential-datalog/pull/977 > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] ovn master test failures
On Wed, May 12, 2021 at 09:21:32PM -0400, Ihar Hrachyshka wrote: > Hi Numan, > > both 3187b9fef124e038e474270a2728fe94bdca8eef (ovn-northd: introduce > new allow-stateless ACL verb) and > 127bf166ccf4a2509f670c48a00b0340039f20d2 (northd: Support flow > offloading for logical switches with no ACLs.) got merged in upstream > master, and this combination broke the following tests: > > 774: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > ovn-northd-ddlog -- dp-groups=yes FAILED (ovn-northd.at:2752) > 775: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > ovn-northd-ddlog FAILED (ovn-northd.at:2752) > > while the other scenarios are passing: > > 768: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > ovn-northd -- dp-groups=yes ok > 769: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- ovn-northd > ok > 770: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > ovn-northd-ddlog -- dp-groups=yes ok > 771: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- > ovn-northd-ddlog ok > 772: ovn -- ACL allow-stateless omit conntrack - Port_Group -- > ovn-northd -- dp-groups=yes ok > 773: ovn -- ACL allow-stateless omit conntrack - Port_Group -- ovn-northd ok This turned to out to be an all-afternoon-and-evening saga, but I got it fixed and documented: https://mail.openvswitch.org/pipermail/ovs-dev/2021-May/382993.html https://github.com/vmware/differential-datalog/pull/977 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] ovn master test failures
Hi Numan,
both 3187b9fef124e038e474270a2728fe94bdca8eef (ovn-northd: introduce
new allow-stateless ACL verb) and
127bf166ccf4a2509f670c48a00b0340039f20d2 (northd: Support flow
offloading for logical switches with no ACLs.) got merged in upstream
master, and this combination broke the following tests:
774: ovn -- ACL allow-stateless omit conntrack - Port_Group --
ovn-northd-ddlog -- dp-groups=yes FAILED (ovn-northd.at:2752)
775: ovn -- ACL allow-stateless omit conntrack - Port_Group --
ovn-northd-ddlog FAILED (ovn-northd.at:2752)
while the other scenarios are passing:
768: ovn -- ACL allow-stateless omit conntrack - Logical_Switch --
ovn-northd -- dp-groups=yes ok
769: ovn -- ACL allow-stateless omit conntrack - Logical_Switch -- ovn-northd ok
770: ovn -- ACL allow-stateless omit conntrack - Logical_Switch --
ovn-northd-ddlog -- dp-groups=yes ok
771: ovn -- ACL allow-stateless omit conntrack - Logical_Switch --
ovn-northd-ddlog ok
772: ovn -- ACL allow-stateless omit conntrack - Port_Group --
ovn-northd -- dp-groups=yes ok
773: ovn -- ACL allow-stateless omit conntrack - Port_Group -- ovn-northd ok
These scenarios (both ok and FAILED) were added with allow-stateless
patch. If I revert "northd: Support flow offloading for logical
switches with no ACLs.", all tests pass.
Two things to note:
1) only ddlog tests fail;
2) only port_group scenarios fail while logical_switch counterparts don't.
Scenarios fail with the following message in testsuite.log:
+2021-05-13T01:06:37Z|1|ovntrace|WARN|lsp1: unknown logical port
+2021-05-13T01:06:37Z|2|ovntrace|WARN|microflow does not specify
ingress port
This is because SB database Port_Binding table is empty when ovn-trace
is executed. In ddlog northd log, I see the following actions:
2021-05-13T01:06:37.902Z|00113|jsonrpc|DBG|unix:/home/ihrachys/dev/ovn/tests/testsuite.dir/774/ovn-nb/ovn-nb.sock:
received notification, method="update",
params=[["monid","OVN_Northbound"],{"Port_Group":{"9ad4d0eb-e643-43e3-aba9-bf80e5349622":{"new":{"ports":["set",[["uuid","04169cef-5ba2-46ee-803c-559e448f9e0e"],["uuid","edd3283e-a056-49d5-9412-6422f01c66df"]]],"name":"pg","external_ids":["map",[]],"acls":["set",[["uuid","02193db0-0b05-4078-a46f-905e12585a22"],["uuid","0277809c-c018-4a35-9100-6f24102ec204"],["uuid","1580f03b-0fb3-4a4b-866b-70c23f8029c5"],["uuid","2a62a6ad-f7f7-42e8-a757-0e5c5215cd0f"],["uuid","2be534fe-8646-4f17-9cec-9cdff8cd1d21"],["uuid","4a714f8c-b901-4ba6-99cd-6150c715f758"],["uuid","7783fd4d-d28f-4636-a191-99c8bd611761"],["uuid","fbfa736e-03d0-4924-9c14-dd3dbc9bb743"]]]},"old":{"acls":["set",[["uuid","02193db0-0b05-4078-a46f-905e12585a22"],["uuid","0277809c-c018-4a35-9100-6f24102ec204"],["uuid","1580f03b-0fb3-4a4b-866b-70c23f8029c5"],["uuid","2a62a6ad-f7f7-42e8-a757-0e5c5215cd0f"],["uuid","2be534fe-8646-4f17-9cec-9cdff8cd1d21"],["uuid","4a714f8c-b901-4ba6-99cd-6150c715f758"],["uuid","7783fd4d-d28f-4636-a191-99c8
bd611761"]]]}}},"ACL":{"fbfa736e-03d0-4924-9c14-dd3dbc9bb743":{"new":{"name":["set",[]],"priority":1,"log":false,"external_ids":["map",[]],"direction":"to-lport","meter":["set",[]],"action":"allow-stateless","match":"tcp","severity":["set",[]]]
2021-05-13T01:06:37.902Z|00114|jsonrpc|DBG|unix:/home/ihrachys/dev/ovn/tests/testsuite.dir/774/ovn-nb/ovn-nb.sock:
received notification, method="update",
params=[["monid","OVN_Northbound"],{"NB_Global":{"ae62228c-aa3d-479a-a251-e612e38e7fdc":{"new":{"name":"","sb_cfg_timestamp":1620867997820,"hv_cfg":1,"nb_cfg":2,"external_ids":["map",[]],"options":["map",[["mac_prefix","06:30:c8"],["max_tunid","16711680"],["northd_internal_version","21.03.90-20.17.0-56.0"],["svc_monitor_mac","c2:cb:ea:d4:18:86"],["use_logical_dp_groups","true"]]],"sb_cfg":1,"ssl":["set",[]],"ipsec":false,"hv_cfg_timestamp":0,"connections":["set",[]],"nb_cfg_timestamp":1620867997730},"old":{"nb_cfg":1]
2021-05-13T01:06:37.934Z|00115|jsonrpc|DBG|unix:/home/ihrachys/dev/ovn/tests/testsuite.dir/774/ovn-sb/ovn-sb.sock:
received reply,
result=[{"uuid":["uuid","1409c320-3e56-2ea6-9f96-ec17e491f2b2"]},{"uuid":["uuid","212b5074-33d6-6f20-d3d4-453b01bb7484"]},{},{}],
id=23
2021-05-13T01:06:37.934Z|00116|jsonrpc|DBG|unix:/home/ihrachys/dev/ovn/tests/testsuite.dir/774/ovn-nb/ovn-nb.sock:
send request, method="transact",
params=["OVN_Northbound",{"where":[["_uuid","==",["uuid","ae62228c-aa3d-479a-a251-e612e38e7fdc"]]],"table":"NB_Global","op":"update","row":{"ipsec":false,"hv_cfg":2,"hv_cfg_timestamp":0,"sb_cfg":1,"options":["map",[["mac_prefix","06:30:c8"],["max_tunid","16711680"],["northd_internal_version","21.03.90-20.17.0-56.0"],["svc_monitor_mac","c2:cb:ea:d4:18:86"],["use_logical_dp_groups","true"]]],"nb_cfg_timestamp":1620867997903}},{"comment":"ovn-northd-ddlog","op":"comment"}],
id=24
2021-05-13T01:06:37.934Z|00117|jsonrpc|DBG|unix:/home/ihrachys/dev/ovn/tests/testsuite.dir/774/ovn-sb/ovn-sb.sock:
send request, method="transact",
