Re: [ovs-dev] ovsdb active backup deployment

[Numan Siddique]

On Sat, Aug 20, 2016 at 2:19 AM, Andy Zhou  wrote:

>
>
> On Fri, Aug 19, 2016 at 11:48 AM, Numan Siddique 
> wrote:
>
>>
>>
>> On Wed, Aug 17, 2016 at 11:24 PM, Andy Zhou  wrote:
>>
>>>
>>>
>>> On Wed, Aug 17, 2016 at 8:30 AM, Numan Siddique 
>>> wrote:
>>>
 ​Hi Andy,
 I have started working on integrating ovsdb-server HA support with
 pacemaker (via OCF heartbeat, i.e ocf:heartbeat).

>>>
>>> Thanks for working on it.
>>>

 Few comments below.



>
> >>> Thanks for helping out.
> >>>
> >>> Given that, I now plan to work from bottom up, initially focusing
> on
> >>> ovsdb server changes.
> >>>
> >>> 1. Add a state in ovsdb-server for it to know whether it is an
> active
> >>> server.  Backup server will not accept any connections.  Server
> started with
> >>> --sync-from argument will be put in the back state by default.
> >>>
> >>> 2. Add appctl commands to allow manually switch state.
> >>>
>

 In order to write the ocf script for ovsdb-server, we need a mechanism
 to
  - know if the ovsdb-server instance is running as master or slave

>>>  Current 2.6 branch code does not have this feature. You can always use
>>> switch commands to
>>> force the state.   On the other hand, adding an appctl command seems
>>> appropriate and can be
>>> useful for trouble shooting as well.  I will work it.
>>>
>>>  - to switch the state of the ovsdb-server to either master or slave.

>>> These are currently supported via appctl commands:
>>> ovsdb-server/connect-active-ovsdb-server
>>> ovsdb-server/disconnect-active-ovsdb-server
>>>
>>
>>
>> ​Thanks for pointing me to the right direction. I added a new command to
>> check if the ovsdb-server is active or backup and submitted the patch [1]
>> as I needed that in the pacemaker ocf script for ovsdb server.
>>
>> [1] - https://patchwork.ozlabs.org/patch/660919/
>>
>
>
>> Based on my initial work, I have a couple of comments below
>>
>> 1. ​
>>
>> ​I want to start the ovsdb servers in 2 nodes in a backup state i.e ​with
>> --sync-with defined for both of them so that pacemaker can promote a master
>> (based on colocation constraints).
>>
> Could the process launch the active ovsdb-server issue an appctl command
> "ovsdb-server/set-active-ovsdb-server"  immediately after.
>
>>
>>   When I start the ovsdb-server's this way, both of them try to get the
>> schema from the other one and get hung. I am not sure if this can be
>> addressed as ovsdb-server is single threaded. Or may be there is a way to
>> address which I am not aware.
>>
> Although that's not an intended use case, I am not aware any reason it
> should hang. I am working a change to replication core to remove blocking
> transactions it uses.
> May be it will help.  It not, I will debug it.
>

​If this issue is resolved, then we don't need to add another command line
option as i had suggested below. It would be good if we can start both the
ovsdb-server's in backup mode ​

​and let ​pacemaker promote one to master.



>>  Hence, I added a new command line option "--backup-server=[true/false]".
>> If back-server is set to true, only then ovsdb-server will try to sync with
>> the active server. This way I can start one server as master (with
>> --sync-from defined) and other as backup (both --sync-from and
>> --backup-server set). It will be easier for pacemaker to demote a master to
>> backup (for what ever reason) so that it can easily connect to the new
>> master (without the need to send the unixctl command -
>> ovsdb-server/set-active-ovsdb-server). I have a patch ready with the new
>> command line option. Please let me know if this is fine.
>>
>
> I don't see why this option is necessary for the given use case. It seems
> the launching process can achieve this by using appctl commands immediately
> after.
>
> On the other hand, if we ever want to make command line option changes,
> this is a good time for it since 2.6 will be the first release to add them.
> A (minor) issue I have with current command line options is that the
> --sync-exclude is not available from the command line.
>
> I considered to consolidating three appctl commands
> set_sync_excluded_tables, set_active_ovsdb_server and
> connect_active_ovsdb_server into a single command, say, sync-from
>   [].  But It seems you don't want
> pacemaker to know about the address of the active/backup server. Is
> this true?
> ​ ​
> What's preventing pacemaker from knowing all the necessary information?
>

​
I am creating a pacemaker resource something like below


pcs resource create tovsdbserver ocf:heartbeat:ovsdb-server
ovsdb_server_unixctl=/usr/local/var/run/openvswitch/ovnsb_db.ctl​ --master
​--​

​This creates a ovsdb-server resource in master/slave mode and it calls the
"start" function of the ovsdb-server OCF script on two nodes of the
cluster. If we want to use unixctrl commands to set

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Fri, Aug 19, 2016 at 11:48 AM, Numan Siddique 
wrote:

>
>
> On Wed, Aug 17, 2016 at 11:24 PM, Andy Zhou  wrote:
>
>>
>>
>> On Wed, Aug 17, 2016 at 8:30 AM, Numan Siddique 
>> wrote:
>>
>>> ​Hi Andy,
>>> I have started working on integrating ovsdb-server HA support with
>>> pacemaker (via OCF heartbeat, i.e ocf:heartbeat).
>>>
>>
>> Thanks for working on it.
>>
>>>
>>> Few comments below.
>>>
>>>
>>>

 >>> Thanks for helping out.
 >>>
 >>> Given that, I now plan to work from bottom up, initially focusing on
 >>> ovsdb server changes.
 >>>
 >>> 1. Add a state in ovsdb-server for it to know whether it is an
 active
 >>> server.  Backup server will not accept any connections.  Server
 started with
 >>> --sync-from argument will be put in the back state by default.
 >>>
 >>> 2. Add appctl commands to allow manually switch state.
 >>>

>>>
>>> In order to write the ocf script for ovsdb-server, we need a mechanism to
>>>  - know if the ovsdb-server instance is running as master or slave
>>>
>>  Current 2.6 branch code does not have this feature. You can always use
>> switch commands to
>> force the state.   On the other hand, adding an appctl command seems
>> appropriate and can be
>> useful for trouble shooting as well.  I will work it.
>>
>>  - to switch the state of the ovsdb-server to either master or slave.
>>>
>> These are currently supported via appctl commands:
>> ovsdb-server/connect-active-ovsdb-server
>> ovsdb-server/disconnect-active-ovsdb-server
>>
>
>
> ​Thanks for pointing me to the right direction. I added a new command to
> check if the ovsdb-server is active or backup and submitted the patch [1]
> as I needed that in the pacemaker ocf script for ovsdb server.
>
> [1] - https://patchwork.ozlabs.org/patch/660919/
>


> Based on my initial work, I have a couple of comments below
>
> 1. ​
>
> ​I want to start the ovsdb servers in 2 nodes in a backup state i.e ​with
> --sync-with defined for both of them so that pacemaker can promote a master
> (based on colocation constraints).
>
Could the process launch the active ovsdb-server issue an appctl command
"ovsdb-server/set-active-ovsdb-server"  immediately after.

>
>   When I start the ovsdb-server's this way, both of them try to get the
> schema from the other one and get hung. I am not sure if this can be
> addressed as ovsdb-server is single threaded. Or may be there is a way to
> address which I am not aware.
>
Although that's not an intended use case, I am not aware any reason it
should hang. I am working a change to replication core to remove blocking
transactions it uses.
May be it will help.  It not, I will debug it.

>
>  Hence, I added a new command line option "--backup-server=[true/false]".
> If back-server is set to true, only then ovsdb-server will try to sync with
> the active server. This way I can start one server as master (with
> --sync-from defined) and other as backup (both --sync-from and
> --backup-server set). It will be easier for pacemaker to demote a master to
> backup (for what ever reason) so that it can easily connect to the new
> master (without the need to send the unixctl command -
> ovsdb-server/set-active-ovsdb-server). I have a patch ready with the new
> command line option. Please let me know if this is fine.
>

I don't see why this option is necessary for the given use case. It seems
the launching process can achieve this by using appctl commands immediately
after.

On the other hand, if we ever want to make command line option changes,
this is a good time for it since 2.6 will be the first release to add them.
A (minor) issue I have with current command line options is that the
--sync-exclude is not available from the command line.

I considered to consolidating three appctl commands
set_sync_excluded_tables, set_active_ovsdb_server and
connect_active_ovsdb_server into a single command, say, sync-from
  [].  But It seems you don't want
pacemaker to know about the address of the active/backup server. Is
this true? What's preventing pacemaker from knowing all the necessary
information?

>
> 2. When the pacemaker has to change the master to backup and backup to
> master (for whatever reason), it first demotes the master and then promotes
> the backup. When it demotes the master, at this point both the
> ovsdb-server's will be in backup modes. After it promotes the backup to
> master, the new backup server is not getting the updates from the new
> master, until I send the unixctl commands - 
> ovsdb-server/disconnect-active-ovsdb-server
> and
> ovsdb-server/connect-active-ovsdb-server
> ​ to the new backup.
> I think this issue needs to be addressed.
>
It looks like another example of issues can be caused by blocking
transactions currently used by replication. Thanks for reporting it. I will
add a test case in my non-blocking patches.  As a work around, could you
relaunch the new backup server after the backup server become active?

>
> ​Thanks
> Numan
>

Re: [ovs-dev] ovsdb active backup deployment

[Valentine Sinitsyn]

On 19.08.2016 23:20, Andy Zhou wrote:



On Fri, Aug 19, 2016 at 1:40 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>> wrote:

On 19.08.2016 02:44, Andy Zhou wrote:



On Thu, Aug 18, 2016 at 1:41 PM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>
>> wrote:

On 18.08.2016 23:49, Andy Zhou wrote:



On Thu, Aug 18, 2016 at 8:34 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>
>


>

>>

>


 wrote:

Hi everyone,

Russell, Would HA manager also
manage
ovn-controller
switch over?


Yes, indirectly.  The way this is
typically
handled
is by
using a virtual
IP that moves to whatever host is
currently
the master

Cool, then ovn-controller does not have
to be HA
aware.


In my understanding, the virtual IP feature in
Pacemaker (i.e.
IPaddr2) works if both active and passive
nodes of the
cluster are
in the same IP subnet.

For some deployments, this would mean both
nodes a
located
on the
same physical rack. This is not actually a
fault-tolerant design
(think blackout).

If I'm getting virtual IP addresses in Pacemaker
correct,
wouldn't
it be better to make ovn-controller HA
aware? That
is, have
a node
switching command (akin to
ovsdb-server/connect-active-ovsdb-server)
and let Pacemaker make use it?


I was not planning to have pacemaker manage
ovn-controller on
every host.

OK, makes sense.

Would using a proxy server, such as HAproxy, help?

Help in what?

To provide a single routable IP address for ovsdb clients to
connect to.

Yes it would, but doesn't Pacemaker handle this already as well?








If this sounds reasonable, I can take on it
probably.


In general, I think having ovn-controller able
to connect to
more than
one database IP seems fine.  I don't expect
everyone to
necessarily
agree on the same HA architecture.

Perhaps it's simple and good enough to add some
support for
multiple IP
addresses for the southbound database that
ovn-controller can rotate
through on reconnect attempts?

As passive ovsdb instance doesn't accept client
connections,
this
wouldn't help much if the connectivity between
ovn-controller and
south ovsdb master is broken. But this scenario is
likely
outside
current HA architecture either.

Re: [ovs-dev] ovsdb active backup deployment

[Numan Siddique]

On Wed, Aug 17, 2016 at 11:24 PM, Andy Zhou  wrote:

>
>
> On Wed, Aug 17, 2016 at 8:30 AM, Numan Siddique 
> wrote:
>
>> ​Hi Andy,
>> I have started working on integrating ovsdb-server HA support with
>> pacemaker (via OCF heartbeat, i.e ocf:heartbeat).
>>
>
> Thanks for working on it.
>
>>
>> Few comments below.
>>
>>
>>
>>>
>>> >>> Thanks for helping out.
>>> >>>
>>> >>> Given that, I now plan to work from bottom up, initially focusing on
>>> >>> ovsdb server changes.
>>> >>>
>>> >>> 1. Add a state in ovsdb-server for it to know whether it is an active
>>> >>> server.  Backup server will not accept any connections.  Server
>>> started with
>>> >>> --sync-from argument will be put in the back state by default.
>>> >>>
>>> >>> 2. Add appctl commands to allow manually switch state.
>>> >>>
>>>
>>
>> In order to write the ocf script for ovsdb-server, we need a mechanism to
>>  - know if the ovsdb-server instance is running as master or slave
>>
>  Current 2.6 branch code does not have this feature. You can always use
> switch commands to
> force the state.   On the other hand, adding an appctl command seems
> appropriate and can be
> useful for trouble shooting as well.  I will work it.
>
>  - to switch the state of the ovsdb-server to either master or slave.
>>
> These are currently supported via appctl commands:
> ovsdb-server/connect-active-ovsdb-server
> ovsdb-server/disconnect-active-ovsdb-server
>


​Thanks for pointing me to the right direction. I added a new command to
check if the ovsdb-server is active or backup and submitted the patch [1]
as I needed that in the pacemaker ocf script for ovsdb server.

[1] - https://patchwork.ozlabs.org/patch/660919/

Based on my initial work, I have a couple of comments below

1. ​

​I want to start the ovsdb servers in 2 nodes in a backup state i.e ​with
--sync-with defined for both of them so that pacemaker can promote a master
(based on colocation constraints).

  When I start the ovsdb-server's this way, both of them try to get the
schema from the other one and get hung. I am not sure if this can be
addressed as ovsdb-server is single threaded. Or may be there is a way to
address which I am not aware.

 Hence, I added a new command line option "--backup-server=[true/false]".
If back-server is set to true, only then ovsdb-server will try to sync with
the active server. This way I can start one server as master (with
--sync-from defined) and other as backup (both --sync-from and
--backup-server set). It will be easier for pacemaker to demote a master to
backup (for what ever reason) so that it can easily connect to the new
master (without the need to send the unixctl command -
ovsdb-server/set-active-ovsdb-server). I have a patch ready with the new
command line option. Please let me know if this is fine.

2. When the pacemaker has to change the master to backup and backup to
master (for whatever reason), it first demotes the master and then promotes
the backup. When it demotes the master, at this point both the
ovsdb-server's will be in backup modes. After it promotes the backup to
master, the new backup server is not getting the updates from the new
master, until I send the unixctl commands -
ovsdb-server/disconnect-active-ovsdb-server and
ovsdb-server/connect-active-ovsdb-server
​ to the new backup.
I think this issue needs to be addressed.

​Thanks
Numan


>  ​
>> The initial (very dirty, not working properly) ocf script  can be found
>> here - [1]
>>
>> I know you have already mentioned about adding this support above. This
>> is just a confirmation that it would be consumed by the pacemaker ocf
>> script once available.
>>
>> [1] - https://github.com/numansiddique/resource-agents/commit/7f
>> 6c4d8977c7cf525ec22793d8adf5b308bc431e
>>
>>
>> ​Thanks
>> Numan
>> ​
>>
>> >>> 3. Add a new table for backup server to register its address and
>>> ports.
>>> >>> OVSDB clients can learn about them at run time. Back up server
>>> should issue
>>> >>> an
>>> >>> transaction to register its address before issuing the monitoring
>>> >>> request.  This feature is not strictly necessary, and can be pushed
>>> to HA
>>> >>> manager,
>>> >>> but having it built into ovsdb-server may make it simpler for
>>> >>> integrationl.
>>> >>>
>>> >>> What do you think?
>>> >>>
>>> >>>
>>> >>>
>>> >> Russell, Would HA manager also manage ovn-controller switch over?
>>> >>
>>> >
>>> > Yes, indirectly.  The way this is typically handled is by using a
>>> virtual
>>> > IP that moves to whatever host is currently the master
>>> >
>>> Cool, then ovn-controller does not have to be HA aware.
>>>
>>> >
>>> >
>>>
>>> >
>>> > --
>>> > Russell Bryant
>>> >
>>> ___
>>> dev mailing list
>>> dev@openvswitch.org
>>> http://openvswitch.org/mailman/listinfo/dev
>>>
>>
>>
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Fri, Aug 19, 2016 at 1:40 AM, Valentine Sinitsyn <
valentine.sinit...@gmail.com> wrote:

> On 19.08.2016 02:44, Andy Zhou wrote:
>
>>
>>
>> On Thu, Aug 18, 2016 at 1:41 PM, Valentine Sinitsyn
>> mailto:valentine.sinit...@gmail.com>>
>> wrote:
>>
>> On 18.08.2016 23:49, Andy Zhou wrote:
>>
>>
>>
>> On Thu, Aug 18, 2016 at 8:34 AM, Valentine Sinitsyn
>> > 
>> > >> wrote:
>>
>> On 18.08.2016 17:42, Russell Bryant wrote:
>>
>>
>> On Thu, Aug 18, 2016 at 2:30 AM, Valentine Sinitsyn
>> > 
>> > >
>> > 
>>
>> > >
>> Hi everyone,
>>
>> Russell, Would HA manager also manage
>> ovn-controller
>> switch over?
>>
>>
>> Yes, indirectly.  The way this is typically
>> handled
>> is by
>> using a virtual
>> IP that moves to whatever host is currently
>> the master
>>
>> Cool, then ovn-controller does not have to be HA
>> aware.
>>
>>
>> In my understanding, the virtual IP feature in
>> Pacemaker (i.e.
>> IPaddr2) works if both active and passive nodes of the
>> cluster are
>> in the same IP subnet.
>>
>> For some deployments, this would mean both nodes a
>> located
>> on the
>> same physical rack. This is not actually a
>> fault-tolerant design
>> (think blackout).
>>
>> If I'm getting virtual IP addresses in Pacemaker
>> correct,
>> wouldn't
>> it be better to make ovn-controller HA aware? That
>> is, have
>> a node
>> switching command (akin to
>> ovsdb-server/connect-active-ovsdb-server)
>> and let Pacemaker make use it?
>>
>>
>> I was not planning to have pacemaker manage
>> ovn-controller on
>> every host.
>>
>> OK, makes sense.
>>
>> Would using a proxy server, such as HAproxy, help?
>>
>> Help in what?
>>
>> To provide a single routable IP address for ovsdb clients to connect to.
>>
> Yes it would, but doesn't Pacemaker handle this already as well?


>
>
>>
>>
>>
>>
>> If this sounds reasonable, I can take on it probably.
>>
>>
>> In general, I think having ovn-controller able to connect
>> to
>> more than
>> one database IP seems fine.  I don't expect everyone to
>> necessarily
>> agree on the same HA architecture.
>>
>> Perhaps it's simple and good enough to add some support
>> for
>> multiple IP
>> addresses for the southbound database that
>> ovn-controller can rotate
>> through on reconnect attempts?
>>
>> As passive ovsdb instance doesn't accept client connections,
>> this
>> wouldn't help much if the connectivity between
>> ovn-controller and
>> south ovsdb master is broken. But this scenario is likely
>> outside
>> current HA architecture either.
>>
>>
>> Yes, something external should change ovsdb-server from backup
>> into
>> active.  A backup server accepts clinet connections, but rejects
>> any
>> "write" transaction that can
>> change the database.
>>
>> Pacamaker is a proposed way to do it, as far as I understand.
>>
>> Right.  I am still in favor of using floating IP for this release.
>>
> Agree. That's a typical was to deploy Pacemaker, and something which would
> work for most people, I guess.
>
> I'm trying to find a solution for a different case, where this IP can
> float within the single rack only (as L3 subnet is not permitted to stretch
> across racks). The mechanisms involved are complementary, not mutually
> exclusive.
>
>
>>
>>
>>
>>
>> In short, yes, having support for multiple IPs in
>> ovn-controller is
>> certainly a step forward in the right direction IMO.
>>
>>
>> I agree it could be a worthwhile feature. If we end up
>> implementing this
>> feature, I hope we don't statically configure the backup server 

Re: [ovs-dev] ovsdb active backup deployment

[Valentine Sinitsyn]

On 19.08.2016 02:44, Andy Zhou wrote:



On Thu, Aug 18, 2016 at 1:41 PM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>> wrote:

On 18.08.2016 23:49, Andy Zhou wrote:



On Thu, Aug 18, 2016 at 8:34 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>
>> wrote:

On 18.08.2016 17:42, Russell Bryant wrote:


On Thu, Aug 18, 2016 at 2:30 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>
>

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Thu, Aug 18, 2016 at 1:41 PM, Valentine Sinitsyn <
valentine.sinit...@gmail.com> wrote:

> On 18.08.2016 23:49, Andy Zhou wrote:
>
>>
>>
>> On Thu, Aug 18, 2016 at 8:34 AM, Valentine Sinitsyn
>> mailto:valentine.sinit...@gmail.com>>
>> wrote:
>>
>> On 18.08.2016 17:42, Russell Bryant wrote:
>>
>>
>> On Thu, Aug 18, 2016 at 2:30 AM, Valentine Sinitsyn
>> > 
>> >
>> >> wrote:
>>
>> Hi everyone,
>>
>> Russell, Would HA manager also manage
>> ovn-controller
>> switch over?
>>
>>
>> Yes, indirectly.  The way this is typically handled
>> is by
>> using a virtual
>> IP that moves to whatever host is currently the master
>>
>> Cool, then ovn-controller does not have to be HA aware.
>>
>>
>> In my understanding, the virtual IP feature in Pacemaker (i.e.
>> IPaddr2) works if both active and passive nodes of the
>> cluster are
>> in the same IP subnet.
>>
>> For some deployments, this would mean both nodes a located
>> on the
>> same physical rack. This is not actually a fault-tolerant
>> design
>> (think blackout).
>>
>> If I'm getting virtual IP addresses in Pacemaker correct,
>> wouldn't
>> it be better to make ovn-controller HA aware? That is, have
>> a node
>> switching command (akin to
>> ovsdb-server/connect-active-ovsdb-server)
>> and let Pacemaker make use it?
>>
>>
>> I was not planning to have pacemaker manage ovn-controller on
>> every host.
>>
>> OK, makes sense.
>>
>> Would using a proxy server, such as HAproxy, help?
>>
> Help in what?
>
> To provide a single routable IP address for ovsdb clients to connect to.

>
>
>>
>>
>> If this sounds reasonable, I can take on it probably.
>>
>>
>> In general, I think having ovn-controller able to connect to
>> more than
>> one database IP seems fine.  I don't expect everyone to
>> necessarily
>> agree on the same HA architecture.
>>
>> Perhaps it's simple and good enough to add some support for
>> multiple IP
>> addresses for the southbound database that ovn-controller can
>> rotate
>> through on reconnect attempts?
>>
>> As passive ovsdb instance doesn't accept client connections, this
>> wouldn't help much if the connectivity between ovn-controller and
>> south ovsdb master is broken. But this scenario is likely outside
>> current HA architecture either.
>>
>>
>> Yes, something external should change ovsdb-server from backup into
>> active.  A backup server accepts clinet connections, but rejects any
>> "write" transaction that can
>> change the database.
>>
> Pacamaker is a proposed way to do it, as far as I understand.

Right.  I am still in favor of using floating IP for this release.

>
>
>
>>
>> In short, yes, having support for multiple IPs in ovn-controller is
>> certainly a step forward in the right direction IMO.
>>
>>
>> I agree it could be a worthwhile feature. If we end up implementing this
>> feature, I hope we don't statically configure the backup server IP
>> address.  It may be better
>> for the idl client to keep track of current backup server.  One possible
>> way to implement it is to store the backup connection into the database.
>>
> Which of the databases? ovn-controller connects to OVS one, and to SB.
> Storing this in OVS means the backup server need to know all
> ovn-controllers on the net, having this information in SB itself is
> somewhat chicken and the egg problem.
>

I'd think we need store them in both DBs.  HA manager or backup server can
update the SB first,  ovn-controller can then replicate the configurations
to its local OVS DB.


> Now we store OVN SB location in the OVS database, do we? My initial intent
> was store not one, but two addresses, and switch between them.
>

I don't object we start there. and this may be fine for the situations
where active and backup server will always occupy two well known addresses.
I just hope we have
a plan to to update those addresses in case HA manager launches the backup
server with a differnt IP address.

>
> Besides, don't we also want ovn-northd to support multiple addresses for
> NB/SB then?


That would be nice too,

>
> The backup server
>> can issue an transaction to record its connection information, before
>> replicating.
>>
>
> Valentine
>
>
>>
>> Best,
>> Valentine
>>
>>
>>
>> --
>> Russell Bryant
>>
>> ___
>> dev mailing list
>> dev@openvswitch.org 
>> http://openvswitch.org/mailman/listinfo/d

Re: [ovs-dev] ovsdb active backup deployment

[Valentine Sinitsyn]

On 18.08.2016 23:49, Andy Zhou wrote:



On Thu, Aug 18, 2016 at 8:34 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>> wrote:

On 18.08.2016 17:42, Russell Bryant wrote:


On Thu, Aug 18, 2016 at 2:30 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>
>> wrote:

Hi everyone,

Russell, Would HA manager also manage ovn-controller
switch over?


Yes, indirectly.  The way this is typically handled
is by
using a virtual
IP that moves to whatever host is currently the master

Cool, then ovn-controller does not have to be HA aware.


In my understanding, the virtual IP feature in Pacemaker (i.e.
IPaddr2) works if both active and passive nodes of the
cluster are
in the same IP subnet.

For some deployments, this would mean both nodes a located
on the
same physical rack. This is not actually a fault-tolerant design
(think blackout).

If I'm getting virtual IP addresses in Pacemaker correct,
wouldn't
it be better to make ovn-controller HA aware? That is, have
a node
switching command (akin to
ovsdb-server/connect-active-ovsdb-server)
and let Pacemaker make use it?


I was not planning to have pacemaker manage ovn-controller on
every host.

OK, makes sense.

Would using a proxy server, such as HAproxy, help?

Help in what?






If this sounds reasonable, I can take on it probably.


In general, I think having ovn-controller able to connect to
more than
one database IP seems fine.  I don't expect everyone to necessarily
agree on the same HA architecture.

Perhaps it's simple and good enough to add some support for
multiple IP
addresses for the southbound database that ovn-controller can rotate
through on reconnect attempts?

As passive ovsdb instance doesn't accept client connections, this
wouldn't help much if the connectivity between ovn-controller and
south ovsdb master is broken. But this scenario is likely outside
current HA architecture either.


Yes, something external should change ovsdb-server from backup into
active.  A backup server accepts clinet connections, but rejects any
"write" transaction that can
change the database.

Pacamaker is a proposed way to do it, as far as I understand.




In short, yes, having support for multiple IPs in ovn-controller is
certainly a step forward in the right direction IMO.


I agree it could be a worthwhile feature. If we end up implementing this
feature, I hope we don't statically configure the backup server IP
address.  It may be better
for the idl client to keep track of current backup server.  One possible
way to implement it is to store the backup connection into the database.
Which of the databases? ovn-controller connects to OVS one, and to SB. 
Storing this in OVS means the backup server need to know all 
ovn-controllers on the net, having this information in SB itself is 
somewhat chicken and the egg problem.


Now we store OVN SB location in the OVS database, do we? My initial 
intent was store not one, but two addresses, and switch between them.


Besides, don't we also want ovn-northd to support multiple addresses for 
NB/SB then?



The backup server
can issue an transaction to record its connection information, before
replicating.


Valentine




Best,
Valentine



--
Russell Bryant

___
dev mailing list
dev@openvswitch.org 
http://openvswitch.org/mailman/listinfo/dev




___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Thu, Aug 18, 2016 at 8:34 AM, Valentine Sinitsyn <
valentine.sinit...@gmail.com> wrote:

> On 18.08.2016 17:42, Russell Bryant wrote:
>
>>
>> On Thu, Aug 18, 2016 at 2:30 AM, Valentine Sinitsyn
>> mailto:valentine.sinit...@gmail.com>>
>> wrote:
>>
>> Hi everyone,
>>
>> Russell, Would HA manager also manage ovn-controller
>> switch over?
>>
>>
>> Yes, indirectly.  The way this is typically handled is by
>> using a virtual
>> IP that moves to whatever host is currently the master
>>
>> Cool, then ovn-controller does not have to be HA aware.
>>
>>
>> In my understanding, the virtual IP feature in Pacemaker (i.e.
>> IPaddr2) works if both active and passive nodes of the cluster are
>> in the same IP subnet.
>>
>> For some deployments, this would mean both nodes a located on the
>> same physical rack. This is not actually a fault-tolerant design
>> (think blackout).
>>
>> If I'm getting virtual IP addresses in Pacemaker correct, wouldn't
>> it be better to make ovn-controller HA aware? That is, have a node
>> switching command (akin to ovsdb-server/connect-active-ovsdb-server)
>> and let Pacemaker make use it?
>>
>>
>> I was not planning to have pacemaker manage ovn-controller on every host.
>>
> OK, makes sense.
>
> Would using a proxy server, such as HAproxy, help?

>
>>
>> If this sounds reasonable, I can take on it probably.
>>
>>
>> In general, I think having ovn-controller able to connect to more than
>> one database IP seems fine.  I don't expect everyone to necessarily
>> agree on the same HA architecture.
>>
>> Perhaps it's simple and good enough to add some support for multiple IP
>> addresses for the southbound database that ovn-controller can rotate
>> through on reconnect attempts?
>>
> As passive ovsdb instance doesn't accept client connections, this wouldn't
> help much if the connectivity between ovn-controller and south ovsdb master
> is broken. But this scenario is likely outside current HA architecture
> either.
>

Yes, something external should change ovsdb-server from backup into
active.  A backup server accepts clinet connections, but rejects any
"write" transaction that can
change the database.

>
> In short, yes, having support for multiple IPs in ovn-controller is
> certainly a step forward in the right direction IMO.
>

I agree it could be a worthwhile feature. If we end up implementing this
feature, I hope we don't statically configure the backup server IP
address.  It may be better
for the idl client to keep track of current backup server.  One possible
way to implement it is to store the backup connection into the database.
The backup server
can issue an transaction to record its connection information, before
replicating.

>
> Best,
> Valentine
>
>
>
>> --
>> Russell Bryant
>>
> ___
> dev mailing list
> dev@openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Valentine Sinitsyn]

On 18.08.2016 17:42, Russell Bryant wrote:


On Thu, Aug 18, 2016 at 2:30 AM, Valentine Sinitsyn
mailto:valentine.sinit...@gmail.com>> wrote:

Hi everyone,

Russell, Would HA manager also manage ovn-controller
switch over?


Yes, indirectly.  The way this is typically handled is by
using a virtual
IP that moves to whatever host is currently the master

Cool, then ovn-controller does not have to be HA aware.


In my understanding, the virtual IP feature in Pacemaker (i.e.
IPaddr2) works if both active and passive nodes of the cluster are
in the same IP subnet.

For some deployments, this would mean both nodes a located on the
same physical rack. This is not actually a fault-tolerant design
(think blackout).

If I'm getting virtual IP addresses in Pacemaker correct, wouldn't
it be better to make ovn-controller HA aware? That is, have a node
switching command (akin to ovsdb-server/connect-active-ovsdb-server)
and let Pacemaker make use it?


I was not planning to have pacemaker manage ovn-controller on every host.

OK, makes sense.




If this sounds reasonable, I can take on it probably.


In general, I think having ovn-controller able to connect to more than
one database IP seems fine.  I don't expect everyone to necessarily
agree on the same HA architecture.

Perhaps it's simple and good enough to add some support for multiple IP
addresses for the southbound database that ovn-controller can rotate
through on reconnect attempts?
As passive ovsdb instance doesn't accept client connections, this 
wouldn't help much if the connectivity between ovn-controller and south 
ovsdb master is broken. But this scenario is likely outside current HA 
architecture either.


In short, yes, having support for multiple IPs in ovn-controller is 
certainly a step forward in the right direction IMO.


Best,
Valentine



--
Russell Bryant

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Russell Bryant]

On Thu, Aug 18, 2016 at 2:30 AM, Valentine Sinitsyn <
valentine.sinit...@gmail.com> wrote:

> Hi everyone,
>
> Russell, Would HA manager also manage ovn-controller switch over?


>>> Yes, indirectly.  The way this is typically handled is by using a virtual
>>> IP that moves to whatever host is currently the master
>>>
>>> Cool, then ovn-controller does not have to be HA aware.
>>
>
> In my understanding, the virtual IP feature in Pacemaker (i.e. IPaddr2)
> works if both active and passive nodes of the cluster are in the same IP
> subnet.
>
> For some deployments, this would mean both nodes a located on the same
> physical rack. This is not actually a fault-tolerant design (think
> blackout).
>
> If I'm getting virtual IP addresses in Pacemaker correct, wouldn't it be
> better to make ovn-controller HA aware? That is, have a node switching
> command (akin to ovsdb-server/connect-active-ovsdb-server) and let
> Pacemaker make use it?
>

I was not planning to have pacemaker manage ovn-controller on every host.


> If this sounds reasonable, I can take on it probably.
>

In general, I think having ovn-controller able to connect to more than one
database IP seems fine.  I don't expect everyone to necessarily agree on
the same HA architecture.

Perhaps it's simple and good enough to add some support for multiple IP
addresses for the southbound database that ovn-controller can rotate
through on reconnect attempts?

-- 
Russell Bryant
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Valentine Sinitsyn]

Hi everyone,


Russell, Would HA manager also manage ovn-controller switch over?



Yes, indirectly.  The way this is typically handled is by using a virtual
IP that moves to whatever host is currently the master


Cool, then ovn-controller does not have to be HA aware.


In my understanding, the virtual IP feature in Pacemaker (i.e. IPaddr2) 
works if both active and passive nodes of the cluster are in the same IP 
subnet.


For some deployments, this would mean both nodes a located on the same 
physical rack. This is not actually a fault-tolerant design (think 
blackout).


If I'm getting virtual IP addresses in Pacemaker correct, wouldn't it be 
better to make ovn-controller HA aware? That is, have a node switching 
command (akin to ovsdb-server/connect-active-ovsdb-server) and let 
Pacemaker make use it?


If this sounds reasonable, I can take on it probably.
If not, I'd greatly appreciate if someone could explain me what I'm missing.

--
Thanks,
Valentine Sinitsyn
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Wed, Aug 17, 2016 at 8:30 AM, Numan Siddique  wrote:

> ​Hi Andy,
> I have started working on integrating ovsdb-server HA support with
> pacemaker (via OCF heartbeat, i.e ocf:heartbeat).
>

Thanks for working on it.

>
> Few comments below.
>
>
>
>>
>> >>> Thanks for helping out.
>> >>>
>> >>> Given that, I now plan to work from bottom up, initially focusing on
>> >>> ovsdb server changes.
>> >>>
>> >>> 1. Add a state in ovsdb-server for it to know whether it is an active
>> >>> server.  Backup server will not accept any connections.  Server
>> started with
>> >>> --sync-from argument will be put in the back state by default.
>> >>>
>> >>> 2. Add appctl commands to allow manually switch state.
>> >>>
>>
>
> In order to write the ocf script for ovsdb-server, we need a mechanism to
>  - know if the ovsdb-server instance is running as master or slave
>
 Current 2.6 branch code does not have this feature. You can always use
switch commands to
force the state.   On the other hand, adding an appctl command seems
appropriate and can be
useful for trouble shooting as well.  I will work it.

 - to switch the state of the ovsdb-server to either master or slave.
>
These are currently supported via appctl commands:
ovsdb-server/connect-active-ovsdb-server
ovsdb-server/disconnect-active-ovsdb-server

 ​
> The initial (very dirty, not working properly) ocf script  can be found
> here - [1]
>
> I know you have already mentioned about adding this support above. This is
> just a confirmation that it would be consumed by the pacemaker ocf script
> once available.
>
> [1] - https://github.com/numansiddique/resource-agents/commit/
> 7f6c4d8977c7cf525ec22793d8adf5b308bc431e
>
>
> ​Thanks
> Numan
> ​
>
> >>> 3. Add a new table for backup server to register its address and ports.
>> >>> OVSDB clients can learn about them at run time. Back up server should
>> issue
>> >>> an
>> >>> transaction to register its address before issuing the monitoring
>> >>> request.  This feature is not strictly necessary, and can be pushed
>> to HA
>> >>> manager,
>> >>> but having it built into ovsdb-server may make it simpler for
>> >>> integrationl.
>> >>>
>> >>> What do you think?
>> >>>
>> >>>
>> >>>
>> >> Russell, Would HA manager also manage ovn-controller switch over?
>> >>
>> >
>> > Yes, indirectly.  The way this is typically handled is by using a
>> virtual
>> > IP that moves to whatever host is currently the master
>> >
>> Cool, then ovn-controller does not have to be HA aware.
>>
>> >
>> >
>>
>> >
>> > --
>> > Russell Bryant
>> >
>> ___
>> dev mailing list
>> dev@openvswitch.org
>> http://openvswitch.org/mailman/listinfo/dev
>>
>
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Numan Siddique]

​Hi Andy,
I have started working on integrating ovsdb-server HA support with
pacemaker (via OCF heartbeat, i.e ocf:heartbeat).

Few comments below.



>
> >>> Thanks for helping out.
> >>>
> >>> Given that, I now plan to work from bottom up, initially focusing on
> >>> ovsdb server changes.
> >>>
> >>> 1. Add a state in ovsdb-server for it to know whether it is an active
> >>> server.  Backup server will not accept any connections.  Server
> started with
> >>> --sync-from argument will be put in the back state by default.
> >>>
> >>> 2. Add appctl commands to allow manually switch state.
> >>>
>

In order to write the ocf script for ovsdb-server, we need a mechanism to
 - know if the ovsdb-server instance is running as master or slave
 - to switch the state of the ovsdb-server to either master or slave.
 ​
The initial (very dirty, not working properly) ocf script  can be found
here - [1]

I know you have already mentioned about adding this support above. This is
just a confirmation that it would be consumed by the pacemaker ocf script
once available.

[1] -
https://github.com/numansiddique/resource-agents/commit/7f6c4d8977c7cf525ec22793d8adf5b308bc431e


​Thanks
Numan
​

>>> 3. Add a new table for backup server to register its address and ports.
> >>> OVSDB clients can learn about them at run time. Back up server should
> issue
> >>> an
> >>> transaction to register its address before issuing the monitoring
> >>> request.  This feature is not strictly necessary, and can be pushed to
> HA
> >>> manager,
> >>> but having it built into ovsdb-server may make it simpler for
> >>> integrationl.
> >>>
> >>> What do you think?
> >>>
> >>>
> >>>
> >> Russell, Would HA manager also manage ovn-controller switch over?
> >>
> >
> > Yes, indirectly.  The way this is typically handled is by using a virtual
> > IP that moves to whatever host is currently the master
> >
> Cool, then ovn-controller does not have to be HA aware.
>
> >
> >
>
> >
> > --
> > Russell Bryant
> >
> ___
> dev mailing list
> dev@openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Thu, Aug 4, 2016 at 11:14 PM, Russell Bryant  wrote:

>
>
> On Thu, Aug 4, 2016 at 8:17 PM, Andy Zhou  wrote:
>
>>
>> On Wed, Jul 27, 2016 at 1:04 PM, Andy Zhou  wrote:
>>
>>>
>>>
>>> On Tue, Jul 26, 2016 at 6:20 PM, Russell Bryant  wrote:
>>>


 On Tue, Jul 26, 2016 at 3:48 PM, Andy Zhou  wrote:

>
>
> On Tue, Jul 26, 2016 at 11:59 AM, Russell Bryant 
> wrote:
>
>>
>>
>> On Tue, Jul 26, 2016 at 2:41 PM, Andy Zhou  wrote:
>>
>>>
>>>
>>> On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant 
>>> wrote:
>>>


 On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:

> Hi, Rayn and Russell,
>

 Can we move this discussion to the ovs dev mailing list?  Feel free
 to just add it in a reply if you'd like.

>>> Done.
>>>


> I am wondering how we can actually use the active/backup feature
> that is now part of
> OVSDB to increase OVN availability.
>

 TO be clear, I haven't actually tried this yet.  I'm only speaking
 about how I think it should work.


> Specifically:
>
> 1. When the active OVSDB server failed, should the back up server
> take over, and allow write transactions? One simpler possibility is to
> allow read only access to the backup serve.
>

 The  backup server needs to take over.  It's OK if that requires
 intervention by an HA manager like Pacemaker.  If we can't make the 
 passive
 server take over, I'd say the solution is incomplete.

>>>
>>> O.K. make sense.
>>>
>>> One possible issue with backup server taking over is "split head".
>>> In case due to network error, backup server becomes disconnected from 
>>> the
>>> active
>>> server, then we may have both server thinking they are active server
>>> now.  Does Pacemaker help with solving this issue.
>>>
>>
>> It can, yes.  I would expect Pacemaker to explicitly configure a node
>> to be either the active or passive node.
>>
> Manual switching is more straight forward. I agree.
>
>>

> 2. When a crashed active OVSDB server recovers, should it become
> the new backup, or it should switch back.
>

 Becoming the new backup is fine.  Again, this can be orchestrated
 by an HA manager (Pacemaker).

>>> I am not familiar with pacemaker. Can I assume it can provide a
>>> correct --sync-from argument (pointing to backup server) when relaunch
>>> OVSDB server?
>>>
>>
>> Yes.  I'd have to consult with some Pacemaker experts on exactly what
>> the implementation would look like, but roughly:
>>
>> Pacemaker manages services using "OCF Resource Agents", which are
>> just scripts with a defined set of inputs and outputs for service
>> management.  I would imagine a Pacemaker cluster being told it must have
>> exactly 1 active and 1 passive OVSDB service.  When the passive OVSDB
>> service is started, it would include the "sync-from" argument based on
>> where the active OVSDB service is currently running.
>>
>> We really need to prototype this and document it.  I'm guessing too
>> much.  Pacemaker is frequently used to manage active/passive HA, though.
>>
>> Sounds reasonable,  I will work on ovsdb internal changes to support
> manual switching, using appctl commands. Then looking into prototyping 
> with
> HA systems.  I have not used pacemaker in the past, so it may take some
> time to ramp up.
>

 I should be able to help.  We need to do this work anyway for
 integration into OpenStack deployment tools.  Let me see if I can get some
 helpful examples to follow.

>>>
>>> Thanks for helping out.
>>>
>>> Given that, I now plan to work from bottom up, initially focusing on
>>> ovsdb server changes.
>>>
>>> 1. Add a state in ovsdb-server for it to know whether it is an active
>>> server.  Backup server will not accept any connections.  Server started with
>>> --sync-from argument will be put in the back state by default.
>>>
>>> 2. Add appctl commands to allow manually switch state.
>>>
>>> 3. Add a new table for backup server to register its address and ports.
>>> OVSDB clients can learn about them at run time. Back up server should issue
>>> an
>>> transaction to register its address before issuing the monitoring
>>> request.  This feature is not strictly necessary, and can be pushed to HA
>>> manager,
>>> but having it built into ovsdb-server may make it simpler for
>>> integrationl.
>>>
>>> What do you think?
>>>
>>>
>>>
>> Russell, Would HA manager also manage ovn-controller switch over?
>>
>
> Yes, indirectly.  The way this is typically handled is by using a virtual
> IP that move

Re: [ovs-dev] ovsdb active backup deployment

[Russell Bryant]

On Thu, Aug 4, 2016 at 8:17 PM, Andy Zhou  wrote:

>
> On Wed, Jul 27, 2016 at 1:04 PM, Andy Zhou  wrote:
>
>>
>>
>> On Tue, Jul 26, 2016 at 6:20 PM, Russell Bryant  wrote:
>>
>>>
>>>
>>> On Tue, Jul 26, 2016 at 3:48 PM, Andy Zhou  wrote:
>>>


 On Tue, Jul 26, 2016 at 11:59 AM, Russell Bryant 
 wrote:

>
>
> On Tue, Jul 26, 2016 at 2:41 PM, Andy Zhou  wrote:
>
>>
>>
>> On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant 
>> wrote:
>>
>>>
>>>
>>> On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:
>>>
 Hi, Rayn and Russell,

>>>
>>> Can we move this discussion to the ovs dev mailing list?  Feel free
>>> to just add it in a reply if you'd like.
>>>
>> Done.
>>
>>>
>>>
 I am wondering how we can actually use the active/backup feature
 that is now part of
 OVSDB to increase OVN availability.

>>>
>>> TO be clear, I haven't actually tried this yet.  I'm only speaking
>>> about how I think it should work.
>>>
>>>
 Specifically:

 1. When the active OVSDB server failed, should the back up server
 take over, and allow write transactions? One simpler possibility is to
 allow read only access to the backup serve.

>>>
>>> The  backup server needs to take over.  It's OK if that requires
>>> intervention by an HA manager like Pacemaker.  If we can't make the 
>>> passive
>>> server take over, I'd say the solution is incomplete.
>>>
>>
>> O.K. make sense.
>>
>> One possible issue with backup server taking over is "split head".
>> In case due to network error, backup server becomes disconnected from the
>> active
>> server, then we may have both server thinking they are active server
>> now.  Does Pacemaker help with solving this issue.
>>
>
> It can, yes.  I would expect Pacemaker to explicitly configure a node
> to be either the active or passive node.
>
 Manual switching is more straight forward. I agree.

>
>>>
 2. When a crashed active OVSDB server recovers, should it become
 the new backup, or it should switch back.

>>>
>>> Becoming the new backup is fine.  Again, this can be orchestrated by
>>> an HA manager (Pacemaker).
>>>
>> I am not familiar with pacemaker. Can I assume it can provide a
>> correct --sync-from argument (pointing to backup server) when relaunch
>> OVSDB server?
>>
>
> Yes.  I'd have to consult with some Pacemaker experts on exactly what
> the implementation would look like, but roughly:
>
> Pacemaker manages services using "OCF Resource Agents", which are just
> scripts with a defined set of inputs and outputs for service management.  
> I
> would imagine a Pacemaker cluster being told it must have exactly 1 active
> and 1 passive OVSDB service.  When the passive OVSDB service is started, 
> it
> would include the "sync-from" argument based on where the active OVSDB
> service is currently running.
>
> We really need to prototype this and document it.  I'm guessing too
> much.  Pacemaker is frequently used to manage active/passive HA, though.
>
> Sounds reasonable,  I will work on ovsdb internal changes to support
 manual switching, using appctl commands. Then looking into prototyping with
 HA systems.  I have not used pacemaker in the past, so it may take some
 time to ramp up.

>>>
>>> I should be able to help.  We need to do this work anyway for
>>> integration into OpenStack deployment tools.  Let me see if I can get some
>>> helpful examples to follow.
>>>
>>
>> Thanks for helping out.
>>
>> Given that, I now plan to work from bottom up, initially focusing on
>> ovsdb server changes.
>>
>> 1. Add a state in ovsdb-server for it to know whether it is an active
>> server.  Backup server will not accept any connections.  Server started with
>> --sync-from argument will be put in the back state by default.
>>
>> 2. Add appctl commands to allow manually switch state.
>>
>> 3. Add a new table for backup server to register its address and ports.
>> OVSDB clients can learn about them at run time. Back up server should issue
>> an
>> transaction to register its address before issuing the monitoring
>> request.  This feature is not strictly necessary, and can be pushed to HA
>> manager,
>> but having it built into ovsdb-server may make it simpler for
>> integrationl.
>>
>> What do you think?
>>
>>
>>
> Russell, Would HA manager also manage ovn-controller switch over?
>

Yes, indirectly.  The way this is typically handled is by using a virtual
IP that moves to whatever host is currently the master.

-- 
Russell Bryant
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Wed, Jul 27, 2016 at 1:04 PM, Andy Zhou  wrote:

>
>
> On Tue, Jul 26, 2016 at 6:20 PM, Russell Bryant  wrote:
>
>>
>>
>> On Tue, Jul 26, 2016 at 3:48 PM, Andy Zhou  wrote:
>>
>>>
>>>
>>> On Tue, Jul 26, 2016 at 11:59 AM, Russell Bryant 
>>> wrote:
>>>


 On Tue, Jul 26, 2016 at 2:41 PM, Andy Zhou  wrote:

>
>
> On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant 
> wrote:
>
>>
>>
>> On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:
>>
>>> Hi, Rayn and Russell,
>>>
>>
>> Can we move this discussion to the ovs dev mailing list?  Feel free
>> to just add it in a reply if you'd like.
>>
> Done.
>
>>
>>
>>> I am wondering how we can actually use the active/backup feature
>>> that is now part of
>>> OVSDB to increase OVN availability.
>>>
>>
>> TO be clear, I haven't actually tried this yet.  I'm only speaking
>> about how I think it should work.
>>
>>
>>> Specifically:
>>>
>>> 1. When the active OVSDB server failed, should the back up server
>>> take over, and allow write transactions? One simpler possibility is to
>>> allow read only access to the backup serve.
>>>
>>
>> The  backup server needs to take over.  It's OK if that requires
>> intervention by an HA manager like Pacemaker.  If we can't make the 
>> passive
>> server take over, I'd say the solution is incomplete.
>>
>
> O.K. make sense.
>
> One possible issue with backup server taking over is "split head".  In
> case due to network error, backup server becomes disconnected from the
> active
> server, then we may have both server thinking they are active server
> now.  Does Pacemaker help with solving this issue.
>

 It can, yes.  I would expect Pacemaker to explicitly configure a node
 to be either the active or passive node.

>>> Manual switching is more straight forward. I agree.
>>>

>>
>>> 2. When a crashed active OVSDB server recovers, should it become the
>>> new backup, or it should switch back.
>>>
>>
>> Becoming the new backup is fine.  Again, this can be orchestrated by
>> an HA manager (Pacemaker).
>>
> I am not familiar with pacemaker. Can I assume it can provide a
> correct --sync-from argument (pointing to backup server) when relaunch
> OVSDB server?
>

 Yes.  I'd have to consult with some Pacemaker experts on exactly what
 the implementation would look like, but roughly:

 Pacemaker manages services using "OCF Resource Agents", which are just
 scripts with a defined set of inputs and outputs for service management.  I
 would imagine a Pacemaker cluster being told it must have exactly 1 active
 and 1 passive OVSDB service.  When the passive OVSDB service is started, it
 would include the "sync-from" argument based on where the active OVSDB
 service is currently running.

 We really need to prototype this and document it.  I'm guessing too
 much.  Pacemaker is frequently used to manage active/passive HA, though.

 Sounds reasonable,  I will work on ovsdb internal changes to support
>>> manual switching, using appctl commands. Then looking into prototyping with
>>> HA systems.  I have not used pacemaker in the past, so it may take some
>>> time to ramp up.
>>>
>>
>> I should be able to help.  We need to do this work anyway for integration
>> into OpenStack deployment tools.  Let me see if I can get some helpful
>> examples to follow.
>>
>
> Thanks for helping out.
>
> Given that, I now plan to work from bottom up, initially focusing on ovsdb
> server changes.
>
> 1. Add a state in ovsdb-server for it to know whether it is an active
> server.  Backup server will not accept any connections.  Server started with
> --sync-from argument will be put in the back state by default.
>
> 2. Add appctl commands to allow manually switch state.
>
> 3. Add a new table for backup server to register its address and ports.
> OVSDB clients can learn about them at run time. Back up server should issue
> an
> transaction to register its address before issuing the monitoring
> request.  This feature is not strictly necessary, and can be pushed to HA
> manager,
> but having it built into ovsdb-server may make it simpler for integrationl.
>
> What do you think?
>
>
>
Russell, Would HA manager also manage ovn-controller switch over?
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Tue, Jul 26, 2016 at 6:20 PM, Russell Bryant  wrote:

>
>
> On Tue, Jul 26, 2016 at 3:48 PM, Andy Zhou  wrote:
>
>>
>>
>> On Tue, Jul 26, 2016 at 11:59 AM, Russell Bryant  wrote:
>>
>>>
>>>
>>> On Tue, Jul 26, 2016 at 2:41 PM, Andy Zhou  wrote:
>>>


 On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant 
 wrote:

>
>
> On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:
>
>> Hi, Rayn and Russell,
>>
>
> Can we move this discussion to the ovs dev mailing list?  Feel free to
> just add it in a reply if you'd like.
>
 Done.

>
>
>> I am wondering how we can actually use the active/backup feature that
>> is now part of
>> OVSDB to increase OVN availability.
>>
>
> TO be clear, I haven't actually tried this yet.  I'm only speaking
> about how I think it should work.
>
>
>> Specifically:
>>
>> 1. When the active OVSDB server failed, should the back up server
>> take over, and allow write transactions? One simpler possibility is to
>> allow read only access to the backup serve.
>>
>
> The  backup server needs to take over.  It's OK if that requires
> intervention by an HA manager like Pacemaker.  If we can't make the 
> passive
> server take over, I'd say the solution is incomplete.
>

 O.K. make sense.

 One possible issue with backup server taking over is "split head".  In
 case due to network error, backup server becomes disconnected from the
 active
 server, then we may have both server thinking they are active server
 now.  Does Pacemaker help with solving this issue.

>>>
>>> It can, yes.  I would expect Pacemaker to explicitly configure a node to
>>> be either the active or passive node.
>>>
>> Manual switching is more straight forward. I agree.
>>
>>>
>
>> 2. When a crashed active OVSDB server recovers, should it become the
>> new backup, or it should switch back.
>>
>
> Becoming the new backup is fine.  Again, this can be orchestrated by
> an HA manager (Pacemaker).
>
 I am not familiar with pacemaker. Can I assume it can provide a correct
 --sync-from argument (pointing to backup server) when relaunch OVSDB
 server?

>>>
>>> Yes.  I'd have to consult with some Pacemaker experts on exactly what
>>> the implementation would look like, but roughly:
>>>
>>> Pacemaker manages services using "OCF Resource Agents", which are just
>>> scripts with a defined set of inputs and outputs for service management.  I
>>> would imagine a Pacemaker cluster being told it must have exactly 1 active
>>> and 1 passive OVSDB service.  When the passive OVSDB service is started, it
>>> would include the "sync-from" argument based on where the active OVSDB
>>> service is currently running.
>>>
>>> We really need to prototype this and document it.  I'm guessing too
>>> much.  Pacemaker is frequently used to manage active/passive HA, though.
>>>
>>> Sounds reasonable,  I will work on ovsdb internal changes to support
>> manual switching, using appctl commands. Then looking into prototyping with
>> HA systems.  I have not used pacemaker in the past, so it may take some
>> time to ramp up.
>>
>
> I should be able to help.  We need to do this work anyway for integration
> into OpenStack deployment tools.  Let me see if I can get some helpful
> examples to follow.
>

Thanks for helping out.

Given that, I now plan to work from bottom up, initially focusing on ovsdb
server changes.

1. Add a state in ovsdb-server for it to know whether it is an active
server.  Backup server will not accept any connections.  Server started with
--sync-from argument will be put in the back state by default.

2. Add appctl commands to allow manually switch state.

3. Add a new table for backup server to register its address and ports.
OVSDB clients can learn about them at run time. Back up server should issue
an
transaction to register its address before issuing the monitoring request.
This feature is not strictly necessary, and can be pushed to HA manager,
but having it built into ovsdb-server may make it simpler for integrationl.

What do you think?



>
> --
> Russell Bryant
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Russell Bryant]

On Tue, Jul 26, 2016 at 3:48 PM, Andy Zhou  wrote:

>
>
> On Tue, Jul 26, 2016 at 11:59 AM, Russell Bryant  wrote:
>
>>
>>
>> On Tue, Jul 26, 2016 at 2:41 PM, Andy Zhou  wrote:
>>
>>>
>>>
>>> On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant  wrote:
>>>


 On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:

> Hi, Rayn and Russell,
>

 Can we move this discussion to the ovs dev mailing list?  Feel free to
 just add it in a reply if you'd like.

>>> Done.
>>>


> I am wondering how we can actually use the active/backup feature that
> is now part of
> OVSDB to increase OVN availability.
>

 TO be clear, I haven't actually tried this yet.  I'm only speaking
 about how I think it should work.


> Specifically:
>
> 1. When the active OVSDB server failed, should the back up server take
> over, and allow write transactions? One simpler possibility is to allow
> read only access to the backup serve.
>

 The  backup server needs to take over.  It's OK if that requires
 intervention by an HA manager like Pacemaker.  If we can't make the passive
 server take over, I'd say the solution is incomplete.

>>>
>>> O.K. make sense.
>>>
>>> One possible issue with backup server taking over is "split head".  In
>>> case due to network error, backup server becomes disconnected from the
>>> active
>>> server, then we may have both server thinking they are active server
>>> now.  Does Pacemaker help with solving this issue.
>>>
>>
>> It can, yes.  I would expect Pacemaker to explicitly configure a node to
>> be either the active or passive node.
>>
> Manual switching is more straight forward. I agree.
>
>>

> 2. When a crashed active OVSDB server recovers, should it become the
> new backup, or it should switch back.
>

 Becoming the new backup is fine.  Again, this can be orchestrated by an
 HA manager (Pacemaker).

>>> I am not familiar with pacemaker. Can I assume it can provide a correct
>>> --sync-from argument (pointing to backup server) when relaunch OVSDB
>>> server?
>>>
>>
>> Yes.  I'd have to consult with some Pacemaker experts on exactly what the
>> implementation would look like, but roughly:
>>
>> Pacemaker manages services using "OCF Resource Agents", which are just
>> scripts with a defined set of inputs and outputs for service management.  I
>> would imagine a Pacemaker cluster being told it must have exactly 1 active
>> and 1 passive OVSDB service.  When the passive OVSDB service is started, it
>> would include the "sync-from" argument based on where the active OVSDB
>> service is currently running.
>>
>> We really need to prototype this and document it.  I'm guessing too
>> much.  Pacemaker is frequently used to manage active/passive HA, though.
>>
>> Sounds reasonable,  I will work on ovsdb internal changes to support
> manual switching, using appctl commands. Then looking into prototyping with
> HA systems.  I have not used pacemaker in the past, so it may take some
> time to ramp up.
>

I should be able to help.  We need to do this work anyway for integration
into OpenStack deployment tools.  Let me see if I can get some helpful
examples to follow.

-- 
Russell Bryant
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Tue, Jul 26, 2016 at 11:59 AM, Russell Bryant  wrote:

>
>
> On Tue, Jul 26, 2016 at 2:41 PM, Andy Zhou  wrote:
>
>>
>>
>> On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant  wrote:
>>
>>>
>>>
>>> On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:
>>>
 Hi, Rayn and Russell,

>>>
>>> Can we move this discussion to the ovs dev mailing list?  Feel free to
>>> just add it in a reply if you'd like.
>>>
>> Done.
>>
>>>
>>>
 I am wondering how we can actually use the active/backup feature that
 is now part of
 OVSDB to increase OVN availability.

>>>
>>> TO be clear, I haven't actually tried this yet.  I'm only speaking about
>>> how I think it should work.
>>>
>>>
 Specifically:

 1. When the active OVSDB server failed, should the back up server take
 over, and allow write transactions? One simpler possibility is to allow
 read only access to the backup serve.

>>>
>>> The  backup server needs to take over.  It's OK if that requires
>>> intervention by an HA manager like Pacemaker.  If we can't make the passive
>>> server take over, I'd say the solution is incomplete.
>>>
>>
>> O.K. make sense.
>>
>> One possible issue with backup server taking over is "split head".  In
>> case due to network error, backup server becomes disconnected from the
>> active
>> server, then we may have both server thinking they are active server
>> now.  Does Pacemaker help with solving this issue.
>>
>
> It can, yes.  I would expect Pacemaker to explicitly configure a node to
> be either the active or passive node.
>
Manual switching is more straight forward. I agree.

>
>>>
 2. When a crashed active OVSDB server recovers, should it become the
 new backup, or it should switch back.

>>>
>>> Becoming the new backup is fine.  Again, this can be orchestrated by an
>>> HA manager (Pacemaker).
>>>
>> I am not familiar with pacemaker. Can I assume it can provide a correct
>> --sync-from argument (pointing to backup server) when relaunch OVSDB
>> server?
>>
>
> Yes.  I'd have to consult with some Pacemaker experts on exactly what the
> implementation would look like, but roughly:
>
> Pacemaker manages services using "OCF Resource Agents", which are just
> scripts with a defined set of inputs and outputs for service management.  I
> would imagine a Pacemaker cluster being told it must have exactly 1 active
> and 1 passive OVSDB service.  When the passive OVSDB service is started, it
> would include the "sync-from" argument based on where the active OVSDB
> service is currently running.
>
> We really need to prototype this and document it.  I'm guessing too much.
> Pacemaker is frequently used to manage active/passive HA, though.
>
> Sounds reasonable,  I will work on ovsdb internal changes to support
manual switching, using appctl commands. Then looking into prototyping with
HA systems.  I have not used pacemaker in the past, so it may take some
time to ramp up.

>
>>>
 Ben said one of you, or both may have worked with similar active-backup
 systems before, so I am very interested in your inputs.

 Thanks,

 Andy

>>>
>>>
>>>
>>> --
>>> Russell Bryant
>>>
>>
>>
> --
> Russell Bryant
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Russell Bryant]

On Tue, Jul 26, 2016 at 2:41 PM, Andy Zhou  wrote:

>
>
> On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant  wrote:
>
>>
>>
>> On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:
>>
>>> Hi, Rayn and Russell,
>>>
>>
>> Can we move this discussion to the ovs dev mailing list?  Feel free to
>> just add it in a reply if you'd like.
>>
> Done.
>
>>
>>
>>> I am wondering how we can actually use the active/backup feature that is
>>> now part of
>>> OVSDB to increase OVN availability.
>>>
>>
>> TO be clear, I haven't actually tried this yet.  I'm only speaking about
>> how I think it should work.
>>
>>
>>> Specifically:
>>>
>>> 1. When the active OVSDB server failed, should the back up server take
>>> over, and allow write transactions? One simpler possibility is to allow
>>> read only access to the backup serve.
>>>
>>
>> The  backup server needs to take over.  It's OK if that requires
>> intervention by an HA manager like Pacemaker.  If we can't make the passive
>> server take over, I'd say the solution is incomplete.
>>
>
> O.K. make sense.
>
> One possible issue with backup server taking over is "split head".  In
> case due to network error, backup server becomes disconnected from the
> active
> server, then we may have both server thinking they are active server now.
> Does Pacemaker help with solving this issue.
>

It can, yes.  I would expect Pacemaker to explicitly configure a node to be
either the active or passive node.


>
>>
>>> 2. When a crashed active OVSDB server recovers, should it become the new
>>> backup, or it should switch back.
>>>
>>
>> Becoming the new backup is fine.  Again, this can be orchestrated by an
>> HA manager (Pacemaker).
>>
> I am not familiar with pacemaker. Can I assume it can provide a correct
> --sync-from argument (pointing to backup server) when relaunch OVSDB
> server?
>

Yes.  I'd have to consult with some Pacemaker experts on exactly what the
implementation would look like, but roughly:

Pacemaker manages services using "OCF Resource Agents", which are just
scripts with a defined set of inputs and outputs for service management.  I
would imagine a Pacemaker cluster being told it must have exactly 1 active
and 1 passive OVSDB service.  When the passive OVSDB service is started, it
would include the "sync-from" argument based on where the active OVSDB
service is currently running.

We really need to prototype this and document it.  I'm guessing too much.
Pacemaker is frequently used to manage active/passive HA, though.


>>
>>> Ben said one of you, or both may have worked with similar active-backup
>>> systems before, so I am very interested in your inputs.
>>>
>>> Thanks,
>>>
>>> Andy
>>>
>>
>>
>>
>> --
>> Russell Bryant
>>
>
>
-- 
Russell Bryant
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] ovsdb active backup deployment

[Andy Zhou]

On Tue, Jul 26, 2016 at 5:37 AM, Russell Bryant  wrote:

>
>
> On Mon, Jul 25, 2016 at 8:15 PM, Andy Zhou  wrote:
>
>> Hi, Rayn and Russell,
>>
>
> Can we move this discussion to the ovs dev mailing list?  Feel free to
> just add it in a reply if you'd like.
>
Done.

>
>
>> I am wondering how we can actually use the active/backup feature that is
>> now part of
>> OVSDB to increase OVN availability.
>>
>
> TO be clear, I haven't actually tried this yet.  I'm only speaking about
> how I think it should work.
>
>
>> Specifically:
>>
>> 1. When the active OVSDB server failed, should the back up server take
>> over, and allow write transactions? One simpler possibility is to allow
>> read only access to the backup serve.
>>
>
> The  backup server needs to take over.  It's OK if that requires
> intervention by an HA manager like Pacemaker.  If we can't make the passive
> server take over, I'd say the solution is incomplete.
>

O.K. make sense.

One possible issue with backup server taking over is "split head".  In case
due to network error, backup server becomes disconnected from the active
server, then we may have both server thinking they are active server now.
Does Pacemaker help with solving this issue.

>
>
>> 2. When a crashed active OVSDB server recovers, should it become the new
>> backup, or it should switch back.
>>
>
> Becoming the new backup is fine.  Again, this can be orchestrated by an HA
> manager (Pacemaker).
>
I am not familiar with pacemaker. Can I assume it can provide a correct
--sync-from argument (pointing to backup server) when relaunch OVSDB
server?

>
>
>> Ben said one of you, or both may have worked with similar active-backup
>> systems before, so I am very interested in your inputs.
>>
>> Thanks,
>>
>> Andy
>>
>
>
>
> --
> Russell Bryant
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev