On 2/6/20 7:39 AM, Vishal Deep Ajmera wrote: >>> >>> So, the root cause of all the issues in this patch, in my understanding, >>> is the fact that you need to collect statistics for all the bond hashes >>> in order to be able to rebalance traffic. This forces you to have access >>> to PMD local caches. >>> >>> The basic idea how to overcome this issue is to not have PMD local bond >>> cache, but have an RCU-protected data structure instead. >>> >>> Memory layout scheme in current patch consists of 3 layers: >>> 1. Global hash map for all bonds. One for the whole datapath. >>> Protected by dp->bond_mutex. >>> 2. Hash map of all the bonds. One for each PMD thread. >>> Protected by pmd->bond_mutex. >>> 3. Hash map of all the bonds. Local copy that could be used >>> lockless, but only by the owning PMD thread. >>> >>> Suggested layout #1: >>> Single global concurrent hash map (cmap) for all bonds. One for the >>> whole >>> datapath. Bond addition/deletion protected by the dp->bond_mutex. >>> Reads are lockless since protected by RCU. Statistics updates must be >>> fully atomic (i.e. atomic_add_relaxed()). >>> >>> Suggested layout #2: >>> One cmap for each PMD thread (no global one). Bond addition/deletion >>> protected by the pmd->bond_mutex. Reads are lockless since protected >>> by RCU. Statistics updates should be atomic in terms of reads and >>> writes. >>> (non_atomic_ullong_add() function could be used). >>> (This is similar to how we store per-PMD flow tables.) >>> >>> #1 will consume a lot less memory, but could scale worse in case of too >>> many >>> threads trying to send traffic to the same bond port. #2 might be a bit >>> faster and more scalable in terms of performance, but less efficient in >>> memory consumption and might be slower in terms of response to slave >> updates >>> since we have to update hash maps on all the threads. >>> Both solutions doesn't require any reconfiguration of running PMD threads. >>> >>> Note that to update cmap entry, you will likely need to prepare the new >>> cmap >>> node and use a cmap_replace(). Statistics copy in this case might be a >>> bit >>> tricky because you may lost part of additions within the period while PMD >>> threads are still using the old entry. To avoid that, statistics copying >>> should be RCU-postponed. However, I'm not sure if we need highly accurate >>> stats there. >>> > > Hi Ilya, > > I am inclined to use option 2 to have per-pmd cmap of bonds however, > I see that even in this option we will need to have one cmap (or hmap) > of bonds at global datapath level. This will take care of > reconfigure_pmd_threads > scenario. New PMDs needs to be configured with bond cmap from global map.
OK. Make sense for a newly created threads. _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
