In scalability test with ovn-scale-test, ovsdb-server SB load is not a problem at least with 1k HVs. However, if we restart the ovsdb-server, depending on the number of HVs and scale of logical objects, e.g. the number of logical ports, ovsdb-server of SB become an obvious bottleneck.
In our test with 1k HVs and 20k logical ports (200 lport * 100 lswitches connected by one single logical router). Restarting ovsdb-server of SB resulted in 100% CPU of ovsdb-server for more than 1 hour. All HVs (and northd) are reconnecting and resyncing the big amount of data at the same time. Similar problem would happen in failover scenario. With active-active cluster, the problem can be aleviated slightly, because only 1/3 (assuming it is 3-node cluster) of the HVs will need to resync data from new servers, but it is still a serious problem. For detailed discussions for the problem and solutions, see: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-October/047591.html The patches implements the proposal in that discussion. It introduces a new method monitor_cond_since to enable client to request changes that happened after a specific point so that the data has been cached already in client are not re-transfered. Scalability test shows dramatic improvement. All HVs finishes sync as soon as they reconnect since there is no new data to be transfered. The current patches supports all 3 modes of ovsdb-server, but only clustered mode can benefit from it, since it is the only one that supports transaction id out of the box. --- v1 -> v2: fix the unused variable in patch 6/7 reported by 0-day. v2 -> v3: first 2 in the 7 were merged. Revised 1/5 addressing Ben's comment on the json cache hmap. v3 -> v4: fix the use after free found by Address Sanitizer in 2/5. Han Zhou (5): ovsdb-monitor: Refactor ovsdb monitor implementation. ovsdb-server: Transaction history tracking. ovsdb-monitor: Support monitor_cond_since. ovsdb-idl.c: Support monitor_cond_since method in C IDL. ovsdb-idl.c: Fast resync from server when connection reset. Documentation/ref/ovsdb-server.7.rst | 78 +++++- lib/ovsdb-idl.c | 229 ++++++++++++---- ovsdb/jsonrpc-server.c | 101 +++++-- ovsdb/monitor.c | 516 ++++++++++++++++++++++------------- ovsdb/monitor.h | 16 +- ovsdb/ovsdb-client.1.in | 28 +- ovsdb/ovsdb-client.c | 104 ++++++- ovsdb/ovsdb-server.c | 9 + ovsdb/ovsdb.c | 6 + ovsdb/ovsdb.h | 10 + ovsdb/transaction.c | 127 ++++++++- ovsdb/transaction.h | 5 + tests/ovsdb-monitor.at | 301 +++++++++++++++++++- 13 files changed, 1244 insertions(+), 286 deletions(-) -- 2.1.0 _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
