On Tue, 25 Aug 2020 at 17:45, Tony Liu <tonyliu0...@hotmail.com> wrote:
>
> Start the first node to create the cluster.
> https://github.com/ovn-org/ovn/blob/master/utilities/ovn-ctl#L228
> https://github.com/openvswitch/ovs/blob/master/utilities/ovs-lib.in#L478
>
> Start the rest nodes to join the cluster.
> https://github.com/ovn-org/ovn/blob/master/utilities/ovn-ctl#L226
> https://github.com/openvswitch/ovs/blob/master/utilities/ovs-lib.in#L478

Unfortunately this is precisely the problem: this doesn't work after
the cluster has already been created. The first node fails to come up
with:

2020-08-26T08:06:19Z|00003|reconnect|INFO|tcp:ovn-ovsdb-1.openstack.svc.cluster.local:6643:
connecting...
2020-08-26T08:06:19Z|00004|reconnect|INFO|tcp:ovn-ovsdb-2.openstack.svc.cluster.local:6643:
connecting...
2020-08-26T08:06:20Z|00005|reconnect|INFO|tcp:ovn-ovsdb-1.openstack.svc.cluster.local:6643:
connection attempt timed out
2020-08-26T08:06:20Z|00006|reconnect|INFO|tcp:ovn-ovsdb-2.openstack.svc.cluster.local:6643:
connection attempt timed out

This makes sense, because the first node can't come up without joining
a quorum, and it can't join a quorum because the other two nodes
aren't up.

I 'fixed' this by switching from the OrderedReady to Parallel pod
management policy for the statefulset. This just means that all pods
come up simultaneously rather than waiting for the first to come up on
its own, which will never work. However, my bootstrapping mechanism
relied on the behaviour of OrderedReady, so I'm going to have to come
up with a solution for that.

Matt

>
> Tony
> > -----Original Message-----
> > From: discuss <ovs-discuss-boun...@openvswitch.org> On Behalf Of Matthew
> > Booth
> > Sent: Tuesday, August 25, 2020 7:08 AM
> > To: ovs-discuss <ovs-discuss@openvswitch.org>
> > Subject: [ovs-discuss] How to restart raft cluster after a complete
> > shutdown?
> >
> > I'm deploying ovsdb-server (and only ovsdb-server) in K8S as a
> > StatefulSet:
> >
> > https://github.com/openstack-k8s-operators/dev-
> > tools/blob/master/ansible/files/ocp/ovn/ovsdb.yaml
> >
> > I'm going to replace this with an operator in due course, which may make
> > the following simpler. I'm not necessarily constrained to only things
> > which are easy to do in a StatefulSet.
> >
> > I've noticed an issue when I kill all 3 pods simultaneously: it is no
> > longer possible to start the cluster. The issue is presumably one of
> > quorum: when a node comes up it can't contact any other node to make
> > quorum, and therefore can't come up. All nodes are similarly affected,
> > so the cluster stays down. Ignoring kubernetes, how is this situation
> > intended to be handled? Do I have to it to a single-node deployment,
> > convert that to a new cluster and re-bootstrap it? This wouldn't be
> > ideal. Is there any way, for example, I can bring up the first node
> > while asserting to that node that the other 2 are definitely down?
> >
> > Thanks,
> >
> > Matt
> > --
> > Matthew Booth
> > Red Hat OpenStack Engineer, Compute DFG
> >
> > Phone: +442070094448 (UK)
> >
> > _______________________________________________
> > discuss mailing list
> > disc...@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>


-- 
Matthew Booth
Red Hat OpenStack Engineer, Compute DFG

Phone: +442070094448 (UK)

_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Reply via email to