Reviewed: https://review.opendev.org/c/openstack/neutron/+/881943 Committed: https://opendev.org/openstack/neutron/commit/c0af5b3b5ea89d3147adf1054625f29d5b01b309 Submitter: "Zuul (22348)" Branch: master
commit c0af5b3b5ea89d3147adf1054625f29d5b01b309 Author: Felix Huettner <[email protected]> Date: Wed Mar 1 16:14:18 2023 +0100 Reduce lock contention on subnets in [1] a lock was introduced with the goal of preventing subnets from being deleted while ports are being created in them in parallel. This was acheived by aquiring an exclusive lock on the row of the subnet in the Subnet table when adding/modifying a port or deleting the subnet. However as this was a exclusive lock it also prevented concurrent port modifications on the same subnet from happening. This can cause performance issues on environment with large shared subnets (e.g. a large external subnet). To reduce the lock contention for this case we split the lock in two parts: * For normal port operations we will aquire a shared lock on the row of the subnet. This allows multiple such operations to happen in parallel. * For deleting a subnet we will aquire an exclusive lock on the row of the subnet. This lock can not be aquired when there is any shared lock currently on the row. With this we maintain the same locking level as before, but reduce the amount of lock contention happening and thereby improve throughput. The performance improvement can be measured using rally test [2]. (improving from 21 to 18 seconds). Alternatively it can be tested using 250 parallel curl calls to create a port in the same network. This improves from 113s to 42s. [1]: https://review.opendev.org/c/openstack/neutron/+/713045 [2]: https://github.com/openstack/rally-openstack/blob/master/samples/tasks/scenarios/neutron/create-and-delete-ports.json Closes-Bug: #2009055 Change-Id: I31b1a9c2f986f59fee0da265acebbd88d2f8e4f8 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2009055 Title: Performance issue when creating lots of ports Status in neutron: Fix Released Bug description: When creating ~250 ports on a single subnet we currently take around 140 seconds. This duration increased significantly with the introduction of a fix for https://bugs.launchpad.net/neutron/+bug/1865891 that exclusively locks a subnet when a port is created/updated in there. This was observed in devstack wih ovn as a backend. Reproducer: 1. Create a network and a subnet 2. Run ``` for i in `seq 1 250`; do curl -g http://10.1.0.129:9696/networking/v2.0/ports -H "Content-Type: application/json" -H "X-Auth-Token: sometoken" -X POST -d '{"port": {"admin_state_up": true, "name": "test", "network_id": "somenetwork"}}' & done time wait ``` We see this causing issues on highly frequented subnets (e.g. the ones hosting our public ips) as the requests take long and die because of our 20 second database max_statement_time. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2009055/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
