On 8/27/25 10:24 PM, Ilya Maximets wrote: > On 8/25/25 10:47 AM, Rukomoinikova Aleksandra wrote: >> Ilya, hi! >> Last time I didnt figured out the issue I wrote about in this thread, >> but I really want to put end to it. I will remind you what I wrote in >> this thread: I applied my patches to version 24.03, these patches shift >> table numbers in northd. After that, ovn userspace test started failing >> (79-82 userspace tests (ECMP symmetric reply). >> The test failed because it expects to see the conntrack state >> ct_state(+new-est-rpl+trk), but after my patches, I see >> ct_state(+new-rpl+trk) in the datapath flow. >> I understand why the userspace test was failing - you said it was all >> due to the classifier's behavior. I figured it out back then and it all >> made sense. I changed the test to expect ct_state(+new-rpl+trk). But >> after that, the test started failing in the kernelspace case. In your >> message above, you said it was likely due to supported features, but I, >> firstly, compared the supported features in the controller logs, and >> secondly, in OpenFlow. They are identical. >> I did some unsuccessful investigation: >> I set up several machines with different distributions. Initially, I >> expected that the test failure might depend on the kernel version. I >> tried different kernels and ended up with the following result: The >> test only fails on Ubuntu 24 - the same is used in ci tests on github >> (which has a 6+ kernel installed). With the same kernel version, the >> test does not fail on CentOS 9, Fedora, or Ubuntu 22.I checked that the >> issue is not related to the GCC version. I compared the kernel configs >> and didn't see anything significant there. I also compared the kernel >> module support. I understand that the issue is likely environmental, >> possibly related to the version of some libraries or system packages. >> But I'm completely out of ideas on where to look next. Maybe you could >> suggest where else I might investigate to get to the bottom of this test >> failure? > > This is very strange... Usually I would suggest to look at hashing. > e.g. if you somehow end up with multiple rules of the same priority > and a random one gets chosen. But it's hard to tell why this can be > if you're using the same compiler flags. Are you using -march=native > or something similar by any chance? Did you try building with -O0
I meant -march=x86-64, but -O0 may also be worth trying just in case. > to avoid any potential instruction set differences? > > It's hard to tell what else can it be. Do you have a link to the > code that you're running? Maybe I can find some time to try it and > see if I can reproduce this behavior. > > Best regards, Ilya Maximets. > >> >> On 18.06.2025 21:41, Ilya Maximets wrote: >>> On 5/26/25 3:45 PM, Rukomoinikova Aleksandra wrote: >>>> Ilya, hi! Thank you for your help and sorry for the long reply. >>>> >>>> Generally, I figured out the classifier code and understood what you >>>> meant, but I noticed that when testing with a kernel module, there's a >>>> difference in the way the classifier works, or something similar. In the >>>> test I was having trouble with, the expected conntrack state is: >>>> +new-est-rpl+trk. After my changes, it turns out to be: +new-rpl+trk. I >>>> corrected the test to expect +new-rpl+trk, but in tests with the OVS >>>> kernel datapath(check kernel), the old сonntrack state is expected: >>>> +new-est-rpl+trk. I couldn't figure out the reason for this behavior, >>>> can you suggest where it would be better to look and what it could be >>>> connected with? Thanks! >>> Sorry, lost track of this discussion for some time. If you run the same >>> test with 'make check-kernel' and the 'make check-system-userspace', do >>> you see different conntrack states? There might be some slight difference >>> in what kind of OpenFlow rules ovn-controller generates depending on the >>> OVS datapath type (features supported). >>> >>>> On 19.05.2025 23:14, Ilya Maximets wrote: >>>>> On 5/16/25 11:16 AM, Rukomoinikova Aleksandra wrote: >>>>>> Hi! >>>>>> >>>>>> We've encountered a strange issue while backporting patches to the >>>>>> version 24.03 branch (ovs v3.3.4) and running tests. Let me describe the >>>>>> situation: >>>>>> I took the upstream branch 24.03, added a stage at the beginning of the >>>>>> switch pipeline, and added a 'match all' flow with 'next;' action. >>>>>> Commit example: >>>>>> https://github.com/Sashhkaa/ovn/commit/f20295315c327addfeb6fe455c3b3c655d6b3666. >>>>>> After this change, OVN 79-82 userspace tests (ECMP symmetric reply) >>>>>> started failing. >>>>>> According to the test logs, I see the following: >>>>>> The test expects to see the conntrack state ct_state(+new-est-rpl+trk) >>>>>> in the datapath flow, but gets ct_state(+new-rpl+trk) - that is, -est >>>>>> disappears. I will also attach more detailed dumps below. >>>>> In general, the extra -est match is harmless and doesn't affect >>>>> correctness, >>>>> because +new traffic is always -est. And +est traffic is always -new. >>>>> So, I think, you may just update the test in your internal backport and >>>>> call it a day. >>>>> >>>>> For the actual reason why this is happening, the answer is: OpenFlow table >>>>> sharing between the switch and the router pipelines. >>>>> >>>>> Both the router and the switch pipelines have their OpenFlow rules in the >>>>> exact same OpenFlow tables starting from table 8. This means that on >>>>> 24.03 >>>>> the ls_in_acl_action and the lr_in_ecmp_stateful stages are using the same >>>>> OpenFlow table 17. When you add one stage to the switch pipeline, you >>>>> shift >>>>> all switch tables by one while keeping router pipelines in place. So, now >>>>> lr_in_ecmp_stateful shares the table with ls_in_acl_eval instead. >>>>> >>>>> All the rules have a match on metadata fields that distinguishes switches >>>>> from routers and so there are no issues with correctness caused by >>>>> sharing. >>>>> However, the classifier may add extra matches due to internal >>>>> implementation >>>>> details. Classifier will traverse all the rules in the OpenFlow table >>>>> starting with the highest priority. If there are no rules that match the >>>>> packet in the current priority, classifier adds a minimal match to the >>>>> datapath flow that will distinguish this packet from any OpenFlow rule in >>>>> this table at this priority. So, if one of the rules with the higher >>>>> priority had +est in the match, classifier will add -est to the datapath >>>>> flow for the packet that didn't match that flow. >>>>> >>>>> So, by adding an extra stage to the router pipeline, you're just restoring >>>>> the mapping of switch and router pipelines to OpenFlow tables like it was >>>>> before the backport. By playing with ACL priorities, you're making the >>>>> classifier go to the next table before evaluating a lower priority rule >>>>> that >>>>> has an extra +est match. >>>>> >>>>> We worked on one similar issue recently: >>>>> >>>>> https://patchwork.ozlabs.org/project/ovn/patch/20250414085122.348614-4-dce...@redhat.com/ >>>>> Here we had a -dnat match leak from the router pipeline to the switch >>>>> pipeline for the packet that does not even go through the router. And >>>>> that >>>>> breaks hardware offload because neither kernel nor hardware NICs support >>>>> offloading of NAT flags. >>>>> >>>>> Leaking of match criteria between switch and router pipelines is an >>>>> interesting side effect of OVN design, but should not generally cause >>>>> issues, >>>>> except for hardware offloading in some cases. >>>>> >>>>> Best regards, Ilya Maximets. >>>>> >>>>>> The expected state should be set by matching this OpenFlow rule in table >>>>>> 17 (in OVN it is router pipeline table 9 - ECMP stateful): >>>>>> >>>>>> cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=6, >>>>>> n_bytes=636, idle_age=1, >>>>>> priority=100,ct_state=+new-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 >>>>>> actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) >>>>>> >>>>>> cookie=0xdda3b0a7, duration=2.635s, table=17, n_packets=14, >>>>>> n_bytes=1396, idle_age=0, >>>>>> priority=100,ct_state=+est-rpl+trk,ipv6,reg14=0x2,metadata=0x1,ipv6_dst=fd01::/126 >>>>>> actions=ct(commit,zone=NXM_NX_REG11[0..15],nat(src),exec(move:NXM_OF_ETH_SRC[]->NXM_NX_CT_LABEL[32..79],load:0x2->NXM_NX_CT_MARK[16..31])),resubmit(,18) >>>>>> >>>>>> >>>>>> I found two logical flow changes, that work, though it's not clear why: >>>>>> 1) Adding a router table before ECMP processing: >>>>>> By inserting just one table at the very beginning of the router >>>>>> pipeline, before the ECMP stateful handling (for example, >>>>>> https://github.com/odivlad/ovn/commit/eb6d0d7409ff78f1fc0908a28225d0a2a47daa29 >>>>>> one table is enough), the test starts passing. The mechanism isn't clear >>>>>> - packets now match the default flow in table 17 and only hit the proper >>>>>> ECMP rule in table 18, yet this somehow resolves the issue. >>>>>> 2) Modifying ACL evaluation rules: >>>>>> The second solution is even more strange. Since this test case doesn't >>>>>> use ACLs or load balancers, northd adds match all' flow with 'next;' >>>>>> action and priority 65535 to the acl_eval table (logical table 9 in >>>>>> switch, OpenFlow table 17). When we lower the priority of these rules >>>>>> below 100(less priority for the ecmp rules), the test begins working. >>>>>> This suggests some hidden interaction between router and switch pipeline >>>>>> rules, despite their different metadata matching criteria. >>>>>> >>>>>> When examining the OVS traces for both cases - the initial failed test >>>>>> with just a stage addition versus the working version where we also >>>>>> modified the ACL eval table priority to 0 - the packet's path through >>>>>> the tables shows no differences except for two key aspects: first, the >>>>>> rule matching in ACL eval (OpenFlow table 17), and second, the resulting >>>>>> datapath action where the -est state unexpectedly disappears. The trace >>>>>> comparison reveals that only the rule priorities in table 17 actually >>>>>> changed, yet this somehow impacts the connection tracking state. You can >>>>>> see the complete trace comparison showing both scenarios - with just the >>>>>> stage addition and with the priority modification - along with the >>>>>> contents of table 17 and the diff between traces at this link: >>>>>> https://gist.github.com/Sashhkaa/58b2c616e7d46fc2dafb898ed832960f. >>>>>> I've verified this behavior persists in newer versions of Open vSwitch >>>>>> as well. >>>>>> Does anyone understand what could be causing this issue? I'd appreciate >>>>>> any insights or suggestions for a proper fix. Thank you! >>>>>> >> > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
