On Wed, Apr 29, 2020 at 10:29 AM Winson Wang <[email protected]> wrote: > > Hello Experts, > > I am doing stress with k8s cluster with ovn, one thing I am seeing is that when raft nodes > got update for large data in short time from ovn-northd, 3 raft nodes will trigger voting and leader role switched from one node to another. > > From ovn-northd side, I can see ovn-northd will trigger the BACKOFF, RECONNECT... > > Since ovn-northd only connect to NB/SB leader only and how can we make ovn-northd more available in most of the time? > > Is it possible to make ovn-northd have established connections to all raft nodes to avoid the > reconnect mechanism? > Since the backoff time 8s is not configurable for now. > > > Test logs: > > 2020-04-29T17:03:08.296Z|41861|ovsdb_idl|INFO|tcp:10.0.2.152:6642: clustered database server is not cluster leader; trying another server > > 2020-04-29T17:03:08.296Z|41862|reconnect|DBG|tcp:10.0.2.152:6642: entering RECONNECT > > 2020-04-29T17:03:08.304Z|41863|reconnect|DBG|tcp:10.0.2.152:6642: entering BACKOFF > > 2020-04-29T17:03:09.708Z|41867|coverage|INFO|Dropped 2 log messages in last 78 seconds (most recently, 71 seconds ago) due to excessive rate > > 2020-04-29T17:03:09.708Z|41868|coverage|INFO|Skipping details of duplicate event coverage for hash=ceada91f > > 2020-04-29T17:03:16.304Z|41869|reconnect|DBG|tcp:10.0.2.153:6642: entering CONNECTING > > 2020-04-29T17:03:16.308Z|41870|reconnect|INFO|tcp:10.0.2.153:6642: connected > > 2020-04-29T17:03:16.308Z|41871|reconnect|DBG|tcp:10.0.2.153:6642: entering ACTIVE > > 2020-04-29T17:03:16.308Z|41872|ovn_northd|INFO|ovn-northd lock lost. This ovn-northd instance is now on standby. > > 2020-04-29T17:03:16.309Z|41873|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active. > > 2020-04-29T17:03:16.311Z|41874|ovsdb_idl|INFO|tcp:10.0.2.153:6642: clustered database server is disconnected from cluster; trying another server > > 2020-04-29T17:03:16.311Z|41875|reconnect|DBG|tcp:10.0.2.153:6642: entering RECONNECT > > 2020-04-29T17:03:16.312Z|41876|reconnect|DBG|tcp:10.0.2.153:6642: entering BACKOFF > > 2020-04-29T17:03:24.316Z|41877|reconnect|DBG|tcp:10.0.2.151:6642: entering CONNECTING > > 2020-04-29T17:03:24.321Z|41878|reconnect|INFO|tcp:10.0.2.151:6642: connected > > 2020-04-29T17:03:24.321Z|41879|reconnect|DBG|tcp:10.0.2.151:6642: entering ACTIVE > > 2020-04-29T17:03:24.321Z|41880|ovn_northd|INFO|ovn-northd lock lost. This ovn-northd instance is now on standby. > > 2020-04-29T17:03:24.354Z|41881|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active. > > 2020-04-29T17:03:24.358Z|41882|ovsdb_idl|INFO|tcp:10.0.2.151:6642: clustered database server is not cluster leader; trying another server > > 2020-04-29T17:03:24.358Z|41883|reconnect|DBG|tcp:10.0.2.151:6642: entering RECONNECT > > 2020-04-29T17:03:24.360Z|41884|reconnect|DBG|tcp:10.0.2.151:6642: entering BACKOFF > > 2020-04-29T17:03:32.367Z|41885|reconnect|DBG|tcp:10.0.2.152:6642: entering CONNECTING > > 2020-04-29T17:03:32.372Z|41886|reconnect|INFO|tcp:10.0.2.152:6642: connected > > 2020-04-29T17:03:32.372Z|41887|reconnect|DBG|tcp:10.0.2.152:6642: entering ACTIVE > > 2020-04-29T17:03:32.372Z|41888|ovn_northd|INFO|ovn-northd lock lost. This ovn-northd instance is now on standby. > > 2020-04-29T17:03:32.373Z|41889|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active. > > 2020-04-29T17:03:32.376Z|41890|ovsdb_idl|INFO|tcp:10.0.2.152:6642: clustered database server is not cluster leader; trying another server > > 2020-04-29T17:03:32.376Z|41891|reconnect|DBG|tcp:10.0.2.152:6642: entering RECONNECT > > 2020-04-29T17:03:32.378Z|41892|reconnect|DBG|tcp:10.0.2.152:6642: entering BACKOFF > > 2020-04-29T17:03:40.381Z|41893|reconnect|DBG|tcp:10.0.2.153:6642: entering CONNECTING > > 2020-04-29T17:03:40.385Z|41894|reconnect|INFO|tcp:10.0.2.153:6642: connected > > 2020-04-29T17:03:40.385Z|41895|reconnect|DBG|tcp:10.0.2.153:6642: entering ACTIVE > > 2020-04-29T17:03:40.385Z|41896|ovn_northd|INFO|ovn-northd lock lost. This ovn-northd instance is now on standby. > > 2020-04-29T17:03:40.385Z|41897|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active. > > > -- > Winson
Hi Winson, Since northd heavily writes to SB DB, it is implemented to connect to leader only, for better performance (avoid the extra cost of a follower forwarding writes to leader). When leader re-election happened, it has to reconnect to the new leader. However, if the cluster is unstable, this step would also take longer time than expected. I'd suggest to tune the election timer to avoid re-election during heavy operations. If the server is overloaded for too long and longer election timer is unacceptable, the only way to solve the availability problem is to improve ovsdb performance. How big is your transaction and what's your election timer setting? The number of clients also impacts the performance since the heavy update needs to be synced to all clients. How many clients do you have? Thanks, Han
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
