On 12/13/21 20:46, Dumitru Ceara wrote:
> Commit 3c2d6274bcee ("raft: Transfer leadership before creating
> snapshots.") made it such that raft leaders transfer leadership before
> snapshotting. However, there's still the case when the next leader to
> be is in the process of snapshotting. To avoid delays in that case too,
> we now explicitly allow snapshots only on followers. Cluster members
> will have to wait until the current election is settled before
> snapshotting.
>
> Given the following logs taken from an OVN_Southbound 3-server cluster
> during a scale test:
>
> S1 (old leader):
> 2021-12-10T19:07:51.226Z|00823|raft|INFO|Transferring leadership to write a
> snapshot.
> 2021-12-10T19:08:03.830Z|00824|ovsdb|INFO|OVN_Southbound: Database
> compaction took 12601ms
> 2021-12-10T19:08:03.833Z|00825|timeval|WARN|Unreasonably long 12604ms poll
> interval (10632ms user, 1924ms system)
> 2021-12-10T19:08:03.940Z|00838|raft|INFO|server 8b8d is leader for term 43
>
> S2 (follower):
> 2021-12-10T19:08:00.870Z|00481|raft|INFO|server 8b8d is leader for term 43
>
> S3 (new leader):
> 2021-12-10T19:07:51.242Z|01083|raft|INFO|received leadership transfer from
> f5c9 in term 42
> 2021-12-10T19:07:51.244Z|01084|raft|INFO|term 43: starting election
> 2021-12-10T19:08:00.805Z|01085|ovsdb|INFO|OVN_Southbound: Database
> compaction took 9559ms
> 2021-12-10T19:08:00.869Z|01100|raft|INFO|term 43: elected leader by 2+ of 3
> servers
>
> We see that the leader to be (S3) receives the leadership transfer,
> initiates the election and immediately after starts a snapshot that
> takes ~9.5 seconds. During this time, S2 votes for S3 electing it
> as cluster leader but S3 doesn't effectively become leader until it
> finishes snapshotting, essentially keeping the cluster without a
> leader for up to ~9.5 seconds.
>
> With the current change, S3 will delay compaction and snapshotting until
> the election is finished.
>
> The only exception is the case of single-node clusters for which we
> allow the node to snapshot regardless of role.
>
> Acked-by: Han Zhou <[email protected]>
> Signed-off-by: Dumitru Ceara <[email protected]>
Thanks, Han and Dumitru! Applied.
Best regards, Ilya Maximets.
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
