On Thu, Jan 25, 2024 at 1:08 AM Han Zhou <[email protected]> wrote:
>
> On Thu, Jan 11, 2024 at 7:32 AM <[email protected]> wrote:
> >
> > From: Numan Siddique <[email protected]>
> >
> > ovn_lflow_add() and other related functions/macros are now moved
> > into a separate module - lflow-mgr.c. This module maintains a
> > table 'struct lflow_table' for the logical flows. lflow table
> > maintains a hmap to store the logical flows.
> >
> > It also maintains the logical switch and router dp groups.
> >
> > Previous commits which added lflow incremental processing for
> > the VIF logical ports, stored the references to
> > the logical ports' lflows using 'struct lflow_ref_list'. This
> > struct is renamed to 'struct lflow_ref' and is part of lflow-mgr.c.
> > It is modified a bit to store the resource to lflow references.
> >
> > Example usage of 'struct lflow_ref'.
> >
> > 'struct ovn_port' maintains 2 instances of lflow_ref. i,e
> >
> > struct ovn_port {
> > ...
> > ...
> > struct lflow_ref *lflow_ref;
> > struct lflow_ref *stateful_lflow_ref;
>
> Hi Numan,
>
> In addition to the lock discussion with you and Dumitru, I still want to
> discuss another thing of this patch regarding the second lflow_ref:
> stateful_lflow_ref.
> I understand that you added this to achieve finer grained I-P especially
> for router ports. I am wondering how much performance gain is from this.
> For my understanding it shouldn't matter much since each ovn_port should be
> associated with a very limited number of lflows. Could you provide more
> insight/data on this? I think it would be better to keep things simple
> (i.e. one object, one lflow_ref list) unless the benefit is obvious.
>
> I am also trying to run another performance regression test for recompute,
> since I am a little concerned about the DP refcnt hmap associated with each
> lflow. I understand it's necessary to handle the duplicated lflow cases,
> but it looks heavy and removes the opportunities for more efficient bitmap
> operations. Let me know if you have already had evaluated its performance.
>
Hi Han,
I did some testing on a large scaled OVN database.
The NB database has
- 1000 logical switches
- 500 routers
- 35253 load balancers
- Most of these load balancers are associated with all the
logical switches and routers.
When I run this command for example
- ovn-nbctl set load_balancer 0d647ff9-4e49-4570-a05d-db670873b7ef
options:foo=bar
It results in changes to all the logical routers. And the function
lflow_handle_lr_stateful_changes() is called.
If you see this function, for each changed router, it also loops
through the router ports and calls
- build_lbnat_lflows_iterate_by_lrp()
- build_lbnat_lflows_iterate_by_lsp()
With your suggestion we also need to call
build_lswitch_and_lrouter_iterate_by_lsp() and
build_lbnat_lflows_iterate_by_lrp().
I measured the number of lflows referenced in op->lflow_ref and
op->stateful_lflow_ref
for each of the logical switch port and router port pair. Total
lflows in lflow_ref
(of all router ports and their peer switch ports) were 23000 and total
lflows in stateful_lflow_ref
were just 4000.
So with just one lflow_ref (as per suggestion) a small update to load
balancer like above
would result in generating 27000 logical flows as compared to just 4000.
I think it has a considerable cost in terms of CPU. And perhaps it
would matter more
when ovn-northd runs in a DPU. My preference would be to have a
separate lflow ref
for stateful flows.
Thanks
Numan
> Thanks,
> Han
>
> > };
> >
> > All the logical flows generated by
> > build_lswitch_and_lrouter_iterate_by_lsp() uses the ovn_port->lflow_ref.
> >
> > All the logical flows generated by build_lsp_lflows_for_lbnats()
> > uses the ovn_port->stateful_lflow_ref.
> >
> > When handling the ovn_port changes incrementally, the lflows referenced
> > in 'struct ovn_port' are cleared and regenerated and synced to the
> > SB logical flows.
> >
> > eg.
> >
> > lflow_ref_clear_lflows(op->lflow_ref);
> > build_lswitch_and_lrouter_iterate_by_lsp(op, ...);
> > lflow_ref_sync_lflows_to_sb(op->lflow_ref, ...);
> >
> > This patch does few more changes:
> > - Logical flows are now hashed without the logical
> > datapaths. If a logical flow is referenced by just one
> > datapath, we don't rehash it.
> >
> > - The synthetic 'hash' column of sbrec_logical_flow now
> > doesn't use the logical datapath. This means that
> > when ovn-northd is updated/upgraded and has this commit,
> > all the logical flows with 'logical_datapath' column
> > set will get deleted and re-added causing some disruptions.
> >
> > - With the commit [1] which added I-P support for logical
> > port changes, multiple logical flows with same match 'M'
> > and actions 'A' are generated and stored without the
> > dp groups, which was not the case prior to
> > that patch.
> > One example to generate these lflows is:
> > ovn-nbctl lsp-set-addresses sw0p1 "MAC1 IP1"
> > ovn-nbctl lsp-set-addresses sw1p1 "MAC1 IP1"
> > ovn-nbctl lsp-set-addresses sw2p1 "MAC1 IP1"
> >
> > Now with this patch we go back to the earlier way. i.e
> > one logical flow with logical_dp_groups set.
> >
> > - With this patch any updates to a logical port which
> > doesn't result in new logical flows will not result in
> > deletion and addition of same logical flows.
> > Eg.
> > ovn-nbctl set logical_switch_port sw0p1 external_ids:foo=bar
> > will be a no-op to the SB logical flow table.
> >
> > [1] - 8bbd678("northd: Incremental processing of VIF additions in 'lflow'
> node.")
> >
> > Signed-off-by: Numan Siddique <[email protected]>
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev