ACK Thanks, Ilya.
For now, update such lock movement code to the test hosts, no cores at sweep__ for 10 days. LIU Yulong On Wed, Mar 20, 2024 at 5:42 AM Ilya Maximets <i.maxim...@ovn.org> wrote: > > On 3/15/24 11:04, LIU Yulong wrote: > > A potential race condition happened with the following 3 threads: > > * PMD thread replaced the old_ukey and transitioned the state to > > UKEY_DELETED. > > * RCU thread is freeing the old_ukey mutex. > > * While the revalidator thread is trying to lock the old_ukey mutex. > > > > We added some timestamp to udpif_key state transition and > > the udpif_key mutex free action, as well as the sweep try lock for > > the same udpif_key. When the ovs-vswitchd goes core, the udpif_key > > try_lock mutex had always a bit later than the udpif_key mutex free > > action. For instance [3]: > > ukey_destroy_time = 13217289156490 > > sweep_now = 13217289156568 > > > > The second time is 78us behind the first time. > > > > Then vswitchd process aborts at the revalidator thread try_lock of > > ukey->mutex because of the NULL pointer. > > > > This patch adds the try_lock for the ukeys' basket udpif_key map to > > avoid the PMD and revalidator access to the same map for replacing the > > udpif_key and transitioning the udpif_key state. > > Thanks for the patch! > > I still need to catch up on the ovs-discuss thread to better understand > what we're dealing with. > > However, the solution proposed here may be a little too radical. The > main problem is that we're holding the mutex for potentially very long > time - the whole iteration over every ukey in the map, potentially > waiting for datapath operations to remove old flows. This can take > hundreds of milliseconds or even a fwe seconds in some cases. For that > time we will be blocking PMD threads and handlers. This is probably > not a good solution. Also, it might be masking the real issue by having > this large critical section, i.e. the issue may still be there, but just > much harder to reproduce. > > Best regards, Ilya Maximets. > _______________________________________________ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
