I’ve re-checked and it really stopped growing after a couple of iterations. Thanks again for the patches.
Regards, Vladislav Odintsov > On 23 Nov 2021, at 02:03, Ilya Maximets <[email protected]> wrote: > > On 11/22/21 20:54, Vladislav Odintsov wrote: >> Hi Ilya, >> >> I’ve tested both patches, the problem with high CPU seems to be solved, >> thanks. >> I noticed see that if I add/delete one lsp to/from port group with a >> negative match rule, ovs-vswitchd each time (add or remove) consumes +4-18 >> MB RSS. >> >> # ovn-nbctl pg-set-ports pg_1 lsp1 lsp2 lsp3 lsp4 >> >> [root@ovn-1 ~]# while :; do ps-mem -p $(pidof ovs-vswitchd) 2> /dev/null | >> grep ovs; sleep 1; done >> 32.6 MiB + 1.4 MiB = 34.0 MiBovs-vswitchd >> 32.6 MiB + 1.4 MiB = 34.0 MiBovs-vswitchd >> 32.6 MiB + 1.4 MiB = 34.0 MiBovs-vswitchd >> 32.5 MiB + 1.4 MiB = 33.8 MiBovs-vswitchd >> 88.7 MiB + 1.4 MiB = 90.1 MiBovs-vswitchd >> 130.7 MiB + 1.4 MiB = 132.1 MiBovs-vswitchd # ovn-nbctl pg-set-ports pg_1 >> lsp1 lsp2 lsp3 >> 130.7 MiB + 1.4 MiB = 132.1 MiBovs-vswitchd >> 130.7 MiB + 1.4 MiB = 132.1 MiBovs-vswitchd >> 130.7 MiB + 1.4 MiB = 132.1 MiBovs-vswitchd >> 130.7 MiB + 1.4 MiB = 132.1 MiBovs-vswitchd >> 130.7 MiB + 1.4 MiB = 132.1 MiBovs-vswitchd >> 141.2 MiB + 1.4 MiB = 142.6 MiBovs-vswitchd # ovn-nbctl pg-set-ports pg_1 >> lsp1 lsp2 lsp3 lsp4 >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 158.4 MiB + 1.4 MiB = 159.8 MiBovs-vswitchd >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd # ovn-nbctl pg-set-ports pg_1 >> lsp1 lsp2 lsp3 >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd >> 162.3 MiB + 1.4 MiB = 163.7 MiBovs-vswitchd >> 170.9 MiB + 1.4 MiB = 172.2 MiBovs-vswitchd # ovn-nbctl pg-set-ports pg_1 >> lsp1 lsp2 lsp3 lsp4 >> 170.9 MiB + 1.4 MiB = 172.2 MiBovs-vswitchd >> 170.9 MiB + 1.4 MiB = 172.2 MiBovs-vswitchd >> 170.9 MiB + 1.4 MiB = 172.2 MiBovs-vswitchd >> >> Can this be also related to above problem? > > I let the script that sets a port group back and forth every 10 seconds > to run for a half an hour while logging the RSS. I'm not familiar with > ps-mem and I don't know how it works, but usual 'ps -aux' showed that > RSS goes up and down between 140 and 170 MB. Values seems to be always > slightly different, even +- few MBs, but RSS stayed within the mentioned > interval. I suppose, this is just how memory allocator works in this > fairly complex case. To me it doesn't look like it's growing continuously, > so should not be a problem. > > Does it continuously grow in your case for extended period of time? > >> >> Regards, >> Vladislav Odintsov >> >>> On 22 Nov 2021, at 18:25, Ilya Maximets <[email protected] >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>>> wrote: >>> >>> On 11/22/21 10:18, Vladislav Odintsov wrote: >>>> Ilya, >>>> >>>> there’s a problem still in place, just with another case. >>>> Initially I’ve tested only creation of topology, but didn’t think about >>>> testing the modification of those flows. >>>> >>>> Create topology from initial mail, and then modify it somehow. For >>>> instance, change the LSPs in port group. >>>> Consider we’ve got lsp1, lsp2, lsp3, lsp4 in pg1 with ACL’s negative >>>> match, then remove lsp4 from pg1 (ovn-nbctl pg-set-ports lsp1 lsp2 lsp3). >>>> Symptoms are the same as it was with initial addition of flows: >>>> high ovn-controller & ovs-vswitchd CPU usage and ovs-vswitchd memory >>>> consumed all memory and then got killed by OOM-killer. >>>> >>>> Let me know if you need any additional info. >>> >>> Thanks for testing! I can reproduce that too. >>> It's a very similar issue, but with a different flavor, so >>> it will require a separate fix. >>> >>> Could you try the following patch together with the previous one: >>> >>> https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/ >>> >>> <https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/><https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/ >>> >>> <https://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/>> >>> ? >>> >>> Bets regards, Ilya Maximets. >>> _______________________________________________ >>> dev mailing list >>> [email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto:[email protected]>> >>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >>> <https://mail.openvswitch.org/mailman/listinfo/ovs-dev> _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
