On 9/2/25 9:29 AM, Han Zhou wrote: > When adding ports to bridges with many existing ports, bridge > reconfiguration was spending significant CPU time in repeated calls to > ofproto_port_open_type() during MTU calculations. > > For N existing ports, this resulted in O(N) calls to > ofproto_port_open_type() per port addition. > > perf shows: > ... > + 75.47% 0.00% ovs-vswitchd ovs-vswitchd [.] ofproto_run > + 66.17% 0.00% ovs-vswitchd ovs-vswitchd [.] update_mtu > + 65.77% 7.27% ovs-vswitchd ovs-vswitchd [.] find_min_mtu > + 65.77% 0.00% ovs-vswitchd ovs-vswitchd [.] update_mtu_ofproto > (inlined) > + 48.75% 1.38% ovs-vswitchd ovs-vswitchd [.] > ofport_is_mtu_overridden > + 48.56% 0.00% ovs-vswitchd ovs-vswitchd [.] > ofport_is_internal_or_patch (inlined) > + 37.19% 2.21% ovs-vswitchd ovs-vswitchd [.] dpif_port_open_type > + 37.19% 0.00% ovs-vswitchd ovs-vswitchd [.] > ofproto_port_open_type (inlined) > ... > > With 5k existing ports, adding a new port took 5.149s. > After this patch, it takes 3.201s for the same.
Hi, Han. Thanks for the patch! I think, it's an interesting change on its own, but I wonder what is the datapath and the port types were in your test? I see a similar performance with dummy datapath, but the reason for that is a not correct behavior of netdev_dummy_set_config() function. It always unconditionally calls netdev_change_seq_changed() and this is actually causing quadratic complexity on the port addition. In the ofproto_run() we collect all the ports that changed into 'devnames' sset and then update_port() is called for each of them that calls update_mtu() then update_mtu_ofproto() and that iterates over all the ports again. So, we always do a nested iteration over all the ports with N^2 complexity. For real ports though this should not normally happen as they will not be marked as changed and will not be added to the 'devnames' set. If I comment out the netdev_change_seq_changed() call, then it takes about 1 second to add a new dummy port at 5000. The time complexity is still quadratic here because of the very inefficient get_port_by_name() implementation in lib/dpif-netdev.c. We don't have a hash map there that would index by name, only by the port number, so this function just iterates over all the ports and compares names. If we add a new hash map there that indexes by name, then the port addition in the dummy datapath (and a general userspace datapath) should be a lot faster. If I do the same test, but adding tunnel ports into a linux kernel datapath, then I do not see the same issue as with dummy ports. One tunnel port at 5000 can be added in about 350ms. This is relatively fast, because we're not doing a lot of syscalls for tunnel ports, since there is only one shared device for all of them in the kernel. The situation is a bit worse for kernel ports that are actually backed by separate kernel interfaces. The main problem here are all the OVS_VPORT_CMD_GET syscalls that are made for each port separately as part of the OFPROTO_PORT_FOR_EACH() macro inside the main iteration in vswitchd/bridge.c:bridge_delete_or_reconfigure_ports(). Addition of one internal port into a linux kernel datapath takes about 500ms at 5000 preexisting ports. I suspect that if we change the logic to dump all the ports once instead of doing a separate syscall per port, this may be done noticeably faster, but I didn't try. All in all, the mtu-related code should not be a real issue, it is only a problem if any time you touch an interface it changes, which is only the case for dummy ports. This should never happen for any type of a real port. What we actually need to optimize: 1. Optimize lib/dpif-netdev.c:get_port_by_name() to avoid the linear port lookup. 2. Re-work OFPROTO_PORT_FOR_EACH() macro to dump all the ports from the datapath once and then look them up in a local hash map, instead of looking pu every port using a separate syscall. 3. Fix the netdev_dummy_set_config() to avoid flagging a change if nothing actually changed. What do you think? Best regards, Ilya Maximets. _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
