Follow-up to [1]: Florian asked for the slot count to come from the
cpu count, Fernando for a way to see the contention. Additional
profiling showed the slot locks were not the main problem, the
per-key list is.

The setup is a 128-cpu hypervisor (64 rx queues) running openvswitch
with per-zone conntrack limits under a 2 Mpps SYN flood against one
VM. OVS keys nf_conncount by zone id, so the whole zone shares one
nf_conncount_list and its list_lock. Once the zone is at its limit
(500k), 96.8% of all cycles are in native_queued_spin_lock_slowpath
behind the spin_lock_bh() in count_tree(), and the host is
unreachable.

Two things add up to that:

 * openvswitch has no way to ask whether the zone has room without
   recording the connection, so above the limit every packet still
   takes list_lock, records an entry that ovs_ct_commit() rejects at
   once, and leaves it for gc to evict;

 * the gc walk stops after 64 *evicted* entries, but live entries and
   -EAGAIN entries do not count, so under flood every walk covers the
   whole list: 500k nf_conntrack_find_get() under list_lock with BH
   off. And nf_conncount_gc_list() takes the trylock before it checks
   whether gc already ran this jiffy, so the count-only lookup is not
   lockless either.

Patch 1, nf_conncount: bound the gc walk by examined entries.
  A pass stops after 128 examined entries and the list is rotated so
  successive passes still cover all of it.

Patch 2, nf_conncount: check last_gc before taking list_lock.
  The count-only lookup stays lockless when gc already ran this
  jiffy; one cpu per jiffy touches the lock, the rest read the count.

Patch 3, nf_conncount: size the number of slots from the cpu count.
  Florian's suggestion from [1].

Patch 4, openvswitch: do not record connections above the zone limit.
  Look the count up first with the existing count-only mode of
  nf_conncount_count_skb() and record only below the limit.

Patch 5, openvswitch: warn about a zone above its limit without a lock.
  The shared state of net_warn_ratelimited() was 2.7% of all cycles.

Each half was also tried alone: patches 1-3 without 4 leave the host
as it was (94% in the slowpath, recording path), patches 4-5 without
1-3 too (97%: the check and the recording are not atomic, every gc
pass that dips the count below the limit lets a burst of cpus record
past it, and the first of them walks the whole list under list_lock).

Numbers, two separate before/after measurements on the same machine.

1. The production case, before and after the whole series:
   openvswitch, zone at its limit, 2 Mpps SYN flood.

                      spinlock slowpath   softirq   host
    unpatched         96%                 100%      unreachable
    patches 1-5       0%                  40%       responsive

2. The nf_conncount part alone, without openvswitch, before and after
   patches 1-3: two netns over a veth pair, iptables "-m connlimit
   --connlimit-above 10000 --connlimit-mask 0" in the receiving one
   (one list, the same shape as a zone), trafgen in the other with 64
   threads sending SYNs from random sources to random ports. veth
   delivers on the sending cpu, so that is 64 cpus on one list_lock.
   Measured with the list full:

                      nf_conncount_count_skb()   calls/s
    unpatched         0.5 - 4 ms                 40k
    patches 1-3       16 - 64 us                 340k

   The lock stays contended here, over 90% of all cycles in the
   slowpath both before and after, because connlimit records every
   connection, see below. What changes is how long the lock is held
   per acquisition, and with it how many packets get through.

The connlimit matches record before they compare, as openvswitch did,
and unlike openvswitch they cannot stop: a match does not know whether
the connection will be dropped, and a softlimit setup needs the
connections above the limit counted. So they keep the convoy, and
patches 1-3 are what helps them.

Tested on the nft side with a ct count test between two netns (limit
enforced, closed and dropped connections reclaimed, list recovers
after being full) under DEBUG_LIST and lockdep.

[1] https://lore.kernel.org/all/[email protected]/

Vladimir Vdovin (5):
  netfilter: nf_conncount: bound the gc walk by examined entries
  netfilter: nf_conncount: check last_gc before taking list_lock in
    gc_list
  netfilter: nf_conncount: size the number of slots from the cpu count
  openvswitch: conntrack: do not record connections above the zone limit
  openvswitch: conntrack: warn about a zone above its limit without a
    lock

 net/netfilter/nf_conncount.c | 86 +++++++++++++++++++++++++++++-------
 net/openvswitch/conntrack.c  | 31 +++++++++++--
 net/openvswitch/datapath.h   |  1 +
 3 files changed, 99 insertions(+), 19 deletions(-)


base-commit: 734bef548a96e8774d6eef5dfa2026c6edc4e80b
-- 
2.56.0

_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Reply via email to