From: Simon Horman <[email protected]> Date: Fri, 6 Sep 2019 19:29:41 +0200
> From: Fred Lotter <[email protected]> > > Flower control message replies are handled in different locations. The truly > high priority replies are handled in the BH (tasklet) context, while the > remaining replies are handled in a predefined Linux work queue. The work > queue handler orders replies into high and low priority groups, and always > start servicing the high priority replies within the received batch first. > > Reply Type: Rtnl Lock: Handler: ... > A subset of control messages can block waiting for an rtnl lock (from both > work queue priority groups). The rtnl lock is heavily contended for by > external processes such as systemd-udevd, systemd-network and libvirtd, > especially during netdev creation, such as when flower VFs and representors > are instantiated. > > Kernel netlink instrumentation shows that external processes (such as > systemd-udevd) often use successive rtnl_trylock() sequences, which can result > in an rtnl_lock() blocked control message to starve for longer periods of time > during rtnl lock contention, i.e. netdev creation. > > In the current design a single blocked control message will block the entire > work queue (both priorities), and introduce a latency which is > nondeterministic and dependent on system wide rtnl lock usage. > > In some extreme cases, one blocked control message at exactly the wrong time, > just before the maximum number of VFs are instantiated, can block the work > queue for long enough to prevent VF representor REIFY replies from getting > handled in time for the 40ms timeout. > > The firmware will deliver the total maximum number of REIFY message replies in > around 300us. > > Only REIFY and MTU update messages require replies within a timeout period (of > 40ms). The MTU-only updates are already done directly in the BH (tasklet) > handler. > > Move the REIFY handler down into the BH (tasklet) in order to resolve timeouts > caused by a blocked work queue waiting on rtnl locks. > > Signed-off-by: Fred Lotter <[email protected]> > Signed-off-by: Simon Horman <[email protected]> Applied.
