On 8/25/25 10:47 AM, Rukomoinikova Aleksandra wrote: > Ilya, hi! > Last time I didnt figured out the issue I wrote about in this thread, > but I really want to put end to it. I will remind you what I wrote in > this thread: I applied my patches to version 24.03, these patches shift > table numbers in northd. After that, ovn userspace test started failing > (79-82 userspace tests (ECMP symmetric reply). > The test failed because it expects to see the conntrack state > ct_state(+new-est-rpl+trk), but after my patches, I see > ct_state(+new-rpl+trk) in the datapath flow. > I understand why the userspace test was failing - you said it was all > due to the classifier's behavior. I figured it out back then and it all > made sense. I changed the test to expect ct_state(+new-rpl+trk). But > after that, the test started failing in the kernelspace case. In your > message above, you said it was likely due to supported features, but I, > firstly, compared the supported features in the controller logs, and > secondly, in OpenFlow. They are identical. > I did some unsuccessful investigation: > I set up several machines with different distributions. Initially, I > expected that the test failure might depend on the kernel version. I > tried different kernels and ended up with the following result: The > test only fails on Ubuntu 24 - the same is used in ci tests on github > (which has a 6+ kernel installed). With the same kernel version, the > test does not fail on CentOS 9, Fedora, or Ubuntu 22.I checked that the > issue is not related to the GCC version. I compared the kernel configs > and didn't see anything significant there. I also compared the kernel > module support. I understand that the issue is likely environmental, > possibly related to the version of some libraries or system packages. > But I'm completely out of ideas on where to look next. Maybe you could > suggest where else I might investigate to get to the bottom of this test > failure?
This is very strange... Usually I would suggest to look at hashing. e.g. if you somehow end up with multiple rules of the same priority and a random one gets chosen. But it's hard to tell why this can be if you're using the same compiler flags. Are you using -march=native or something similar by any chance? Did you try building with -O0 to avoid any potential instruction set differences? It's hard to tell what else can it be. Do you have a link to the code that you're running? Maybe I can find some time to try it and see if I can reproduce this behavior. Best regards, Ilya Maximets. > > On 18.06.2025 21:41, Ilya Maximets wrote: >> On 5/26/25 3:45 PM, Rukomoinikova Aleksandra wrote: >>> Ilya, hi! Thank you for your help and sorry for the long reply. >>> >>> Generally, I figured out the classifier code and understood what you >>> meant, but I noticed that when testing with a kernel module, there's a >>> difference in the way the classifier works, or something similar. In the >>> test I was having trouble with, the expected conntrack state is: >>> +new-est-rpl+trk. After my changes, it turns out to be: +new-rpl+trk. I >>> corrected the test to expect +new-rpl+trk, but in tests with the OVS >>> kernel datapath(check kernel), the old сonntrack state is expected: >>> +new-est-rpl+trk. I couldn't figure out the reason for this behavior, >>> can you suggest where it would be better to look and what it could be >>> connected with? Thanks! >> Sorry, lost track of this discussion for some time. If you run the same >> test with 'make check-kernel' and the 'make check-system-userspace', do >> you see different conntrack states? There might be some slight difference >> in what kind of OpenFlow rules ovn-controller generates depending on the >> OVS datapath type (features supported). >> >>> On 19.05.2025 23:14, Ilya Maximets wrote: >>>> On 5/16/25 11:16 AM, Rukomoinikova Aleksandra wrote: >>>>> Hi! >>>>> >>>>> We've encountered a strange issue while backporting patches to the >>>>> version 24.03 branch (ovs v3.3.4) and running tests. Let me describe the >>>>> situation: >>>>> I took the upstream branch 24.03, added a stage at the beginning of the >>>>> switch pipeline, and added a 'match all' flow with 'next;' action. >>>>> Commit example: >>>>> https://github.com/Sashhkaa/ovn/commit/f20295315c327addfeb6fe455c3b3c655d6b3666. >>>>> After this change, OVN 79-82 userspace tests (ECMP symmetric reply) >>>>> started failing. >>>>> According to the test logs, I see the following: >>>>> The test expects to see the conntrack state ct_state(+new-est-rpl+trk) >>>>> in the datapath flow, but gets ct_state(+new-rpl+trk) - that is, -est >>>>> disappears. I will also attach more detailed dumps below. >>>> In general, the extra -est match is harmless and doesn't affect >>>> correctness, >>>> because +new traffic is always -est. And +est traffic is always -new. >>>> So, I think, you may just update the test in your internal backport and >>>> call it a day. >>>> >>>> For the actual reason why this is happening, the answer is: OpenFlow table >>>> sharing between the switch and the router pipelines. >>>> >>>> Both the router and the switch pipelines have their OpenFlow rules in the >>>> exact same OpenFlow tables starting from table 8. This means that on 24.03 >>>> the ls_in_acl_action and the lr_in_ecmp_stateful stages are using the same >>>> OpenFlow table 17. When you add one stage to the switch pipeline, you >>>> shift >>>> all switch tables by one while keeping router pipelines in place. So, now >>>> lr_in_ecmp_stateful shares the table with ls_in_acl_eval instead. >>>> >>>> All the rules have a match on metadata fields that distinguishes switches >>>> from routers and so there are no issues with correctness caused by sharing. >>>> However, the classifier may add extra matches due to internal >>>> implementation >>>> details. Classifier will traverse all the rules in the OpenFlow table >>>> starting with the highest priority. If there are no rules that match the >>>> packet in the current priority, classifier adds a minimal match to the >>>> datapath flow that will distinguish this packet from any OpenFlow rule in >>>> this table at this priority. So, if one of the rules with the higher >>>> priority had +est in the match, classifier will add -est to the datapath >>>> flow for the packet that didn't match that flow. >>>> >>>> So, by adding an extra stage to the router pipeline, you're just restoring >>>> the mapping of switch and router pipelines to OpenFlow tables like it was >>>> before the backport. By playing with ACL priorities, you're making the >>>> classifier go to the next table before evaluating a lower priority rule >>>> that >>>> has an extra +est match. >>>> >>>> We worked on one similar issue recently: >>>> >>>> https://patchwork.ozlabs.org/project/ovn/patch/20250414085122.348614-4-dce...@redhat.com/ >>>> Here we had a -dnat match leak from the router pipeline to the switch >>>> pipeline for the packet that does not even go through the router. And that >>>> breaks hardware offload because neither kernel nor hardware NICs support >>>> offloading of NAT flags. >>>> >>>> Leaking of match criteria between switch and router pipelines is an >>>> interesting side effect of OVN design, but should not generally cause >>>> issues, >>>> except for hardware offloading in some cases. >>>> >>>> Best regards, Ilya Maximets. >>>> >>>>> The expected state should be set by matching this OpenFlow rule in table >>>>> 17 (in OVN it is router pipeline table 9 - ECMP stateful): >>>>> >>>>> cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=6, >>>>> n_bytes=636, idle_age=1, >>>>> priority=100,ct_state=+new-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 >>>>> actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) >>>>> >>>>> cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=14, >>>>> n_bytes=1396, idle_age=0, >>>>> priority=100,ct_state=+est-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 >>>>> actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) >>>>> >>>>> >>>>> I found two logical flow changes, that work, though it's not clear why: >>>>> 1) Adding a router table before ECMP processing: >>>>> By inserting just one table at the very beginning of the router >>>>> pipeline, before the ECMP stateful handling (for example, >>>>> https://github.com/odivlad/ovn/commit/eb6d0d7409ff78f1fc0908a28225d0a2a47daa29 >>>>> one table is enough), the test starts passing. The mechanism isn't clear >>>>> - packets now match the default flow in table 17 and only hit the proper >>>>> ECMP rule in table 18, yet this somehow resolves the issue. >>>>> 2) Modifying ACL evaluation rules: >>>>> The second solution is even more strange. Since this test case doesn't >>>>> use ACLs or load balancers, northd adds match all' flow with 'next;' >>>>> action and priority 65535 to the acl_eval table (logical table 9 in >>>>> switch, OpenFlow table 17). When we lower the priority of these rules >>>>> below 100(less priority for the ecmp rules), the test begins working. >>>>> This suggests some hidden interaction between router and switch pipeline >>>>> rules, despite their different metadata matching criteria. >>>>> >>>>> When examining the OVS traces for both cases - the initial failed test >>>>> with just a stage addition versus the working version where we also >>>>> modified the ACL eval table priority to 0 - the packet's path through >>>>> the tables shows no differences except for two key aspects: first, the >>>>> rule matching in ACL eval (OpenFlow table 17), and second, the resulting >>>>> datapath action where the -est state unexpectedly disappears. The trace >>>>> comparison reveals that only the rule priorities in table 17 actually >>>>> changed, yet this somehow impacts the connection tracking state. You can >>>>> see the complete trace comparison showing both scenarios - with just the >>>>> stage addition and with the priority modification - along with the >>>>> contents of table 17 and the diff between traces at this link: >>>>> https://gist.github.com/Sashhkaa/58b2c616e7d46fc2dafb898ed832960f. >>>>> I've verified this behavior persists in newer versions of Open vSwitch >>>>> as well. >>>>> Does anyone understand what could be causing this issue? I'd appreciate >>>>> any insights or suggestions for a proper fix. Thank you! >>>>> > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
