Hi! We are experiencing high latencies from time to time in one of our openstack environments. While looking into the issue we found that the southdb has frequent leader changes.
We are running the Xena release and ovsdb-server (Open vSwitch) 2.16.0 deployed via kolla-ansible. --- cut --- 2022-04-07T06:51:41.782Z|00952|raft|INFO|received leadership transfer from 3da8 in term 7545 2022-04-07T07:10:27.993Z|00925|raft|INFO|received leadership transfer from 1c5f in term 7546 2022-04-07T07:10:53.355Z|01061|raft|INFO|received leadership transfer from f177 in term 7547 2022-04-07T07:28:09.193Z|00938|raft|INFO|received leadership transfer from 1c5f in term 7548 2022-04-07T07:43:24.377Z|01166|raft|INFO|received leadership transfer from f177 in term 7549 2022-04-07T07:44:01.979Z|01031|raft|INFO|received leadership transfer from 1c5f in term 7550 2022-04-07T08:02:17.876Z|01179|raft|INFO|received leadership transfer from f177 in term 7551 2022-04-07T08:15:21.329Z|01137|raft|INFO|received leadership transfer from 1c5f in term 7552 2022-04-07T08:32:54.440Z|01262|raft|INFO|received leadership transfer from f177 in term 7553 2022-04-07T08:34:05.864Z|01248|raft|INFO|received leadership transfer from 1c5f in term 7554 2022-04-07T08:47:48.497Z|01279|raft|INFO|received leadership transfer from f177 in term 7555 2022-04-07T08:48:12.508Z|01333|raft|INFO|received leadership transfer from 1c5f in term 7556 2022-04-07T09:03:08.086Z|01286|raft|INFO|received leadership transfer from f177 in term 7557 2022-04-07T09:03:27.895Z|01428|raft|INFO|received leadership transfer from 1c5f in term 7558 2022-04-07T09:16:34.045Z|01291|raft|INFO|received leadership transfer from f177 in term 7559 2022-04-07T09:25:54.219Z|01512|raft|INFO|received leadership transfer from 1c5f in term 7560 2022-04-07T09:32:59.026Z|01348|raft|INFO|received leadership transfer from f177 in term 7561 2022-04-07T09:42:53.125Z|01556|raft|INFO|received leadership transfer from 1c5f in term 7562 2022-04-07T09:46:42.450Z|01414|raft|INFO|received leadership transfer from f177 in term 7563 2022-04-07T09:54:20.665Z|01582|raft|INFO|received leadership transfer from 1c5f in term 7564 2022-04-07T09:58:28.906Z|01458|raft|INFO|received leadership transfer from f177 in term 7565 2022-04-07T10:09:09.271Z|01610|raft|INFO|received leadership transfer from 1c5f in term 7566 2022-04-07T10:26:56.792Z|01528|raft|INFO|received leadership transfer from f177 in term 7567 2022-04-07T10:27:34.885Z|01716|raft|INFO|received leadership transfer from 1c5f in term 7568 2022-04-07T10:41:38.178Z|01533|raft|INFO|received leadership transfer from f177 in term 7569 2022-04-07T10:42:25.543Z|01807|raft|INFO|received leadership transfer from 1c5f in term 7570 2022-04-07T10:55:46.772Z|01540|raft|INFO|received leadership transfer from f177 in term 7571 2022-04-07T11:00:55.610Z|01892|raft|INFO|received leadership transfer from 1c5f in term 7572 2022-04-07T11:14:17.057Z|01575|raft|INFO|received leadership transfer from f177 in term 7573 2022-04-07T11:26:58.164Z|01971|raft|INFO|received leadership transfer from 1c5f in term 7574 2022-04-07T11:43:44.300Z|01657|raft|INFO|received leadership transfer from f177 in term 7575 2022-04-07T11:53:17.514Z|02075|raft|INFO|received leadership transfer from 1c5f in term 7576 2022-04-07T11:59:42.938Z|01714|raft|INFO|received leadership transfer from f177 in term 7577 2022-04-07T12:07:32.316Z|02121|raft|INFO|received leadership transfer from 1c5f in term 7578 2022-04-07T12:18:19.280Z|01765|raft|INFO|received leadership transfer from f177 in term 7579 2022-04-07T12:22:57.588Z|02186|raft|INFO|received leadership transfer from 1c5f in term 7580 2022-04-07T12:30:09.755Z|01799|raft|INFO|received leadership transfer from f177 in term 7581 2022-04-07T12:36:53.283Z|02233|raft|INFO|received leadership transfer from 1c5f in term 7582 2022-04-07T12:44:09.022Z|01846|raft|INFO|received leadership transfer from f177 in term 7583 2022-04-07T12:47:16.956Z|02278|raft|INFO|received leadership transfer from 1c5f in term 7584 2022-04-07T12:54:14.197Z|01870|raft|INFO|received leadership transfer from f177 in term 7585 --- cut --- We also noticed that the leader changes are caused by snapshots (database compaction?). --- cut --- 2022-04-07T07:10:27.993Z|01057|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T07:28:09.193Z|01162|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T07:44:01.979Z|01175|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T08:15:21.329Z|01258|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T08:34:05.863Z|01275|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T08:48:12.508Z|01282|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T09:03:27.894Z|01289|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T09:25:54.219Z|01344|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T09:42:53.125Z|01410|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T09:54:20.665Z|01454|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T10:09:09.270Z|01524|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T10:27:34.884Z|01531|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T10:42:25.543Z|01536|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T11:00:55.610Z|01571|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T11:26:58.164Z|01653|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T11:53:17.514Z|01710|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T12:07:32.316Z|01759|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T12:22:57.588Z|01795|raft|INFO|Transferring leadership to write a snapshot. 2022-04-07T12:36:53.282Z|01842|raft|INFO|Transferring leadership to write a snapshot. --- cut --- Do these changes/snapshots have an impact on the performance of the southdb? Is there anything to improve the performance in case of leadership changes? Kind regards, Christian Stelter _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
