On Mon, Jan 29, 2024 at 7:11 PM Numan Siddique <[email protected]> wrote:
> On Thu, Jan 25, 2024 at 1:08 AM Han Zhou <[email protected]> wrote: > > > > On Thu, Jan 11, 2024 at 7:32 AM <[email protected]> wrote: > > > > > > From: Numan Siddique <[email protected]> > > > > > > ovn_lflow_add() and other related functions/macros are now moved > > > into a separate module - lflow-mgr.c. This module maintains a > > > table 'struct lflow_table' for the logical flows. lflow table > > > maintains a hmap to store the logical flows. > > > > > > It also maintains the logical switch and router dp groups. > > > > > > Previous commits which added lflow incremental processing for > > > the VIF logical ports, stored the references to > > > the logical ports' lflows using 'struct lflow_ref_list'. This > > > struct is renamed to 'struct lflow_ref' and is part of lflow-mgr.c. > > > It is modified a bit to store the resource to lflow references. > > > > > > Example usage of 'struct lflow_ref'. > > > > > > 'struct ovn_port' maintains 2 instances of lflow_ref. i,e > > > > > > struct ovn_port { > > > ... > > > ... > > > struct lflow_ref *lflow_ref; > > > struct lflow_ref *stateful_lflow_ref; > > > > Hi Numan, > > > > In addition to the lock discussion with you and Dumitru, I still want to > > discuss another thing of this patch regarding the second lflow_ref: > > stateful_lflow_ref. > > I understand that you added this to achieve finer grained I-P especially > > for router ports. I am wondering how much performance gain is from this. > > For my understanding it shouldn't matter much since each ovn_port should > be > > associated with a very limited number of lflows. Could you provide more > > insight/data on this? I think it would be better to keep things simple > > (i.e. one object, one lflow_ref list) unless the benefit is obvious. > > > > I am also trying to run another performance regression test for > recompute, > > since I am a little concerned about the DP refcnt hmap associated with > each > > lflow. I understand it's necessary to handle the duplicated lflow cases, > > but it looks heavy and removes the opportunities for more efficient > bitmap > > operations. Let me know if you have already had evaluated its > performance. > > > > Hi Han, > > I did some testing on a large scaled OVN database. > The NB database has > - 1000 logical switches > - 500 routers > - 35253 load balancers > - Most of these load balancers are associated with all the > logical switches and routers. > > When I run this command for example > - ovn-nbctl set load_balancer 0d647ff9-4e49-4570-a05d-db670873b7ef > options:foo=bar > > It results in changes to all the logical routers. And the function > lflow_handle_lr_stateful_changes() is called. > If you see this function, for each changed router, it also loops > through the router ports and calls > - build_lbnat_lflows_iterate_by_lrp() > - build_lbnat_lflows_iterate_by_lsp() > > With your suggestion we also need to call > build_lswitch_and_lrouter_iterate_by_lsp() and > build_lbnat_lflows_iterate_by_lrp(). > > I measured the number of lflows referenced in op->lflow_ref and > op->stateful_lflow_ref > for each of the logical switch port and router port pair. Total > lflows in lflow_ref > (of all router ports and their peer switch ports) were 23000 and total > lflows in stateful_lflow_ref > were just 4000. > > So with just one lflow_ref (as per suggestion) a small update to load > balancer like above > would result in generating 27000 logical flows as compared to just 4000. > > I think it has a considerable cost in terms of CPU. And perhaps it > would matter more > when ovn-northd runs in a DPU. My preference would be to have a > separate lflow ref > for stateful flows. > > Thanks > Numan Thanks for the data points! It makes sense to use separate lists. Regards, Han > > > > > Thanks, > > Han > > > > > }; > > > > > > All the logical flows generated by > > > build_lswitch_and_lrouter_iterate_by_lsp() uses the > ovn_port->lflow_ref. > > > > > > All the logical flows generated by build_lsp_lflows_for_lbnats() > > > uses the ovn_port->stateful_lflow_ref. > > > > > > When handling the ovn_port changes incrementally, the lflows referenced > > > in 'struct ovn_port' are cleared and regenerated and synced to the > > > SB logical flows. > > > > > > eg. > > > > > > lflow_ref_clear_lflows(op->lflow_ref); > > > build_lswitch_and_lrouter_iterate_by_lsp(op, ...); > > > lflow_ref_sync_lflows_to_sb(op->lflow_ref, ...); > > > > > > This patch does few more changes: > > > - Logical flows are now hashed without the logical > > > datapaths. If a logical flow is referenced by just one > > > datapath, we don't rehash it. > > > > > > - The synthetic 'hash' column of sbrec_logical_flow now > > > doesn't use the logical datapath. This means that > > > when ovn-northd is updated/upgraded and has this commit, > > > all the logical flows with 'logical_datapath' column > > > set will get deleted and re-added causing some disruptions. > > > > > > - With the commit [1] which added I-P support for logical > > > port changes, multiple logical flows with same match 'M' > > > and actions 'A' are generated and stored without the > > > dp groups, which was not the case prior to > > > that patch. > > > One example to generate these lflows is: > > > ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1" > > > ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1" > > > ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1" > > > > > > Now with this patch we go back to the earlier way. i.e > > > one logical flow with logical_dp_groups set. > > > > > > - With this patch any updates to a logical port which > > > doesn't result in new logical flows will not result in > > > deletion and addition of same logical flows. > > > Eg. > > > ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar > > > will be a no-op to the SB logical flow table. > > > > > > [1] - 8bbd678("northd: Incremental processing of VIF additions in > 'lflow' > > node.") > > > > > > Signed-off-by: Numan Siddique <[email protected]> > > _______________________________________________ > > dev mailing list > > [email protected] > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
