On 5/16/25 11:16 AM, Rukomoinikova Aleksandra wrote: > Hi! > > We've encountered a strange issue while backporting patches to the > version 24.03 branch (ovs v3.3.4) and running tests. Let me describe the > situation: > I took the upstream branch 24.03, added a stage at the beginning of the > switch pipeline, and added a 'match all' flow with 'next;' action. > Commit example: > https://github.com/Sashhkaa/ovn/commit/f20295315c327addfeb6fe455c3b3c655d6b3666. > > After this change, OVN 79-82 userspace tests (ECMP symmetric reply) > started failing. > According to the test logs, I see the following: > The test expects to see the conntrack state ct_state(+new-est-rpl+trk) > in the datapath flow, but gets ct_state(+new-rpl+trk) - that is, -est > disappears. I will also attach more detailed dumps below.
In general, the extra -est match is harmless and doesn't affect correctness, because +new traffic is always -est. And +est traffic is always -new. So, I think, you may just update the test in your internal backport and call it a day. For the actual reason why this is happening, the answer is: OpenFlow table sharing between the switch and the router pipelines. Both the router and the switch pipelines have their OpenFlow rules in the exact same OpenFlow tables starting from table 8. This means that on 24.03 the ls_in_acl_action and the lr_in_ecmp_stateful stages are using the same OpenFlow table 17. When you add one stage to the switch pipeline, you shift all switch tables by one while keeping router pipelines in place. So, now lr_in_ecmp_stateful shares the table with ls_in_acl_eval instead. All the rules have a match on metadata fields that distinguishes switches from routers and so there are no issues with correctness caused by sharing. However, the classifier may add extra matches due to internal implementation details. Classifier will traverse all the rules in the OpenFlow table starting with the highest priority. If there are no rules that match the packet in the current priority, classifier adds a minimal match to the datapath flow that will distinguish this packet from any OpenFlow rule in this table at this priority. So, if one of the rules with the higher priority had +est in the match, classifier will add -est to the datapath flow for the packet that didn't match that flow. So, by adding an extra stage to the router pipeline, you're just restoring the mapping of switch and router pipelines to OpenFlow tables like it was before the backport. By playing with ACL priorities, you're making the classifier go to the next table before evaluating a lower priority rule that has an extra +est match. We worked on one similar issue recently: https://patchwork.ozlabs.org/project/ovn/patch/20250414085122.348614-4-dce...@redhat.com/ Here we had a -dnat match leak from the router pipeline to the switch pipeline for the packet that does not even go through the router. And that breaks hardware offload because neither kernel nor hardware NICs support offloading of NAT flags. Leaking of match criteria between switch and router pipelines is an interesting side effect of OVN design, but should not generally cause issues, except for hardware offloading in some cases. Best regards, Ilya Maximets. > > The expected state should be set by matching this OpenFlow rule in table > 17 (in OVN it is router pipeline table 9 - ECMP stateful): > > cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=6, > n_bytes=636, idle_age=1, > priority=100,ct_state=+new-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 > > actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) > > > cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=14, > n_bytes=1396, idle_age=0, > priority=100,ct_state=+est-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 > > actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) > > > > I found two logical flow changes, that work, though it's not clear why: > 1) Adding a router table before ECMP processing: > By inserting just one table at the very beginning of the router > pipeline, before the ECMP stateful handling (for example, > https://github.com/odivlad/ovn/commit/eb6d0d7409ff78f1fc0908a28225d0a2a47daa29 > > one table is enough), the test starts passing. The mechanism isn't clear > - packets now match the default flow in table 17 and only hit the proper > ECMP rule in table 18, yet this somehow resolves the issue. > 2) Modifying ACL evaluation rules: > The second solution is even more strange. Since this test case doesn't > use ACLs or load balancers, northd adds match all' flow with 'next;' > action and priority 65535 to the acl_eval table (logical table 9 in > switch, OpenFlow table 17). When we lower the priority of these rules > below 100(less priority for the ecmp rules), the test begins working. > This suggests some hidden interaction between router and switch pipeline > rules, despite their different metadata matching criteria. > > When examining the OVS traces for both cases - the initial failed test > with just a stage addition versus the working version where we also > modified the ACL eval table priority to 0 - the packet's path through > the tables shows no differences except for two key aspects: first, the > rule matching in ACL eval (OpenFlow table 17), and second, the resulting > datapath action where the -est state unexpectedly disappears. The trace > comparison reveals that only the rule priorities in table 17 actually > changed, yet this somehow impacts the connection tracking state. You can > see the complete trace comparison showing both scenarios - with just the > stage addition and with the priority modification - along with the > contents of table 17 and the diff between traces at this link: > https://gist.github.com/Sashhkaa/58b2c616e7d46fc2dafb898ed832960f. > I've verified this behavior persists in newer versions of Open vSwitch > as well. > Does anyone understand what could be causing this issue? I'd appreciate > any insights or suggestions for a proper fix. Thank you! > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
