Re: [ovs-dev] [replication SMv2 7/7] ovsdb: Replication usability improvements

[Andy Zhou]

>
> I tested the v3 version pulling from your github, and I am not seeing an
>> crashes.
>>
>
>
> Thanks for testing. I added your name to tested-by in the V3 patch.
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [replication SMv2 7/7] ovsdb: Replication usability improvements

[Numan Siddique]

On Wed, Aug 31, 2016 at 5:36 PM, Numan Siddique  wrote:

>
>
> On Wed, Aug 31, 2016 at 12:03 AM, Andy Zhou  wrote:
>
>>
>>
>> On Tue, Aug 30, 2016 at 4:17 AM, Numan Siddique 
>> wrote:
>>
>>>
>>>
>>> On Tue, Aug 30, 2016 at 1:11 AM, Andy Zhou  wrote:
>>>


 On Mon, Aug 29, 2016 at 3:14 AM, Numan Siddique 
 wrote:

>
>
> On Sat, Aug 27, 2016 at 4:45 AM, Andy Zhou  wrote:
>
>> Added the '--no-sync' option base on feedbacks of current
>> implementation.
>>
>> Added appctl command "ovsdb-server/sync-status" based on feedbacks
>> of current implementation.
>>
>> Added a test to simulate the integration of HA manager with OVSDB
>> server using replication.
>>
>> Other documentation and API improvements.
>>
>> Signed-off-by: Andy Zhou 
>> --
>>
>> I hope to get some review comments on the command line and appctl
>> interfaces for replication. Since 2.6 is the first release of those
>> interfaces, it is easier to making changes, compare to future
>> releases.
>>
>> 
>> v1->v2: Fix creashes reported at:
>> http://openvswitch.org/pipermail/dev/2016-August/078591.html
>> ---
>>
>
> ​I haven't tested these patches yet. This patch seems to have a white
> space warning when applied.
>
 Thanks for the reported. I will fold the fix in the next version when
 posting.

 In case it helps, you can also access the patches from my private repo
 at:
   https://github.com/azhou-nicira/ovs-review/tree/ovsdb-replic
 ation-sm-v2


>>> ​
>>> Hi Andy,
>>> ​
>>> I am seeing the below crash when
>>>
>>> ​  - The ovsdb-server changes from
>>> ​master to standby and the active-ovsdb-server it is about to connect to
>>> is killed just before that or it is not reachable.
>>>
>>> ​  -
>>> ​The pacemaker OCF script calls the sync-status cmd soon after that.
>>>
>>>
>>> ​Please let me know if you need more information.​
>>>
>>>
>>> ​Core was generated by `ovsdb-server -vdbg 
>>> --log-file=/opt/stack/logs/ovsdb-server-sb.log
>>> --remote=puni'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>> #0  0x0041241d in replication_status () at
>>> ovsdb/replication.c:875
>>> 875SHASH_FOR_EACH (node, replication_dbs) {
>>> Missing separate debuginfos, use: dnf debuginfo-install
>>> glibc-2.23.1-10.fc24.x86_64 openssl-libs-1.0.2h-3.fc24.x86_64
>>> (gdb) bt
>>> #0  0x0041241d in replication_status () at
>>> ovsdb/replication.c:875
>>> #1  0x00406eda in ovsdb_server_get_sync_status (conn=0x1421fd0,
>>> argc=, argv=, config_=)
>>> at ovsdb/ovsdb-server.c:1480
>>> #2  0x004324ee in process_command (request=0x1421f30,
>>> conn=0x1421fd0) at lib/unixctl.c:313
>>> #3  run_connection (conn=0x1421fd0) at lib/unixctl.c:347
>>> #4  unixctl_server_run (server=server@entry=0x141e140) at
>>> lib/unixctl.c:400
>>> #5  0x00405bdc in main_loop (is_backup=0x7fff08062256,
>>> exiting=0x7fff08062257, run_process=0x0, remotes=0x7fff080622a0,
>>> unixctl=0x141e140,
>>> all_dbs=0x7fff080622e0, jsonrpc=0x13f6f00) at
>>> ovsdb/ovsdb-server.c:182
>>> #6  main (argc=, argv=) at
>>> ovsdb/ovsdb-server.c:430​
>>>
>>> Numan, thanks for the report. I think I spotted the bug:
>>
>> Currently, when replication state machine is reset,  the state update
>> takes place after a round of main loop run. this time lag
>> could lead to the back trace in case the unixctl commands was issued
>> during this time lag.  I have a fix that add another
>> state to represent the reset condition.  The fix is at:
>>
>> https://github.com/azhou-nicira/ovs-review/tree/ovsdb-replication-sm-v3
>>
>> Would you please let me know if this version works any better?. Thanks!
>>
>
> ​Sure. I would test and let you know.
>
>
​I tested the v3 version pulling from your github, and I am not seeing an
crashes.

Thanks
Numan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [replication SMv2 7/7] ovsdb: Replication usability improvements

[Numan Siddique]

On Wed, Aug 31, 2016 at 12:03 AM, Andy Zhou  wrote:

>
>
> On Tue, Aug 30, 2016 at 4:17 AM, Numan Siddique 
> wrote:
>
>>
>>
>> On Tue, Aug 30, 2016 at 1:11 AM, Andy Zhou  wrote:
>>
>>>
>>>
>>> On Mon, Aug 29, 2016 at 3:14 AM, Numan Siddique 
>>> wrote:
>>>


 On Sat, Aug 27, 2016 at 4:45 AM, Andy Zhou  wrote:

> Added the '--no-sync' option base on feedbacks of current
> implementation.
>
> Added appctl command "ovsdb-server/sync-status" based on feedbacks
> of current implementation.
>
> Added a test to simulate the integration of HA manager with OVSDB
> server using replication.
>
> Other documentation and API improvements.
>
> Signed-off-by: Andy Zhou 
> --
>
> I hope to get some review comments on the command line and appctl
> interfaces for replication. Since 2.6 is the first release of those
> interfaces, it is easier to making changes, compare to future
> releases.
>
> 
> v1->v2: Fix creashes reported at:
> http://openvswitch.org/pipermail/dev/2016-August/078591.html
> ---
>

 ​I haven't tested these patches yet. This patch seems to have a white
 space warning when applied.

>>> Thanks for the reported. I will fold the fix in the next version when
>>> posting.
>>>
>>> In case it helps, you can also access the patches from my private repo
>>> at:
>>>   https://github.com/azhou-nicira/ovs-review/tree/ovsdb-replic
>>> ation-sm-v2
>>>
>>>
>> ​
>> Hi Andy,
>> ​
>> I am seeing the below crash when
>>
>> ​  - The ovsdb-server changes from
>> ​master to standby and the active-ovsdb-server it is about to connect to
>> is killed just before that or it is not reachable.
>>
>> ​  -
>> ​The pacemaker OCF script calls the sync-status cmd soon after that.
>>
>>
>> ​Please let me know if you need more information.​
>>
>>
>> ​Core was generated by `ovsdb-server -vdbg 
>> --log-file=/opt/stack/logs/ovsdb-server-sb.log
>> --remote=puni'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  0x0041241d in replication_status () at ovsdb/replication.c:875
>> 875SHASH_FOR_EACH (node, replication_dbs) {
>> Missing separate debuginfos, use: dnf debuginfo-install
>> glibc-2.23.1-10.fc24.x86_64 openssl-libs-1.0.2h-3.fc24.x86_64
>> (gdb) bt
>> #0  0x0041241d in replication_status () at ovsdb/replication.c:875
>> #1  0x00406eda in ovsdb_server_get_sync_status (conn=0x1421fd0,
>> argc=, argv=, config_=)
>> at ovsdb/ovsdb-server.c:1480
>> #2  0x004324ee in process_command (request=0x1421f30,
>> conn=0x1421fd0) at lib/unixctl.c:313
>> #3  run_connection (conn=0x1421fd0) at lib/unixctl.c:347
>> #4  unixctl_server_run (server=server@entry=0x141e140) at
>> lib/unixctl.c:400
>> #5  0x00405bdc in main_loop (is_backup=0x7fff08062256,
>> exiting=0x7fff08062257, run_process=0x0, remotes=0x7fff080622a0,
>> unixctl=0x141e140,
>> all_dbs=0x7fff080622e0, jsonrpc=0x13f6f00) at ovsdb/ovsdb-server.c:182
>> #6  main (argc=, argv=) at
>> ovsdb/ovsdb-server.c:430​
>>
>> Numan, thanks for the report. I think I spotted the bug:
>
> Currently, when replication state machine is reset,  the state update
> takes place after a round of main loop run. this time lag
> could lead to the back trace in case the unixctl commands was issued
> during this time lag.  I have a fix that add another
> state to represent the reset condition.  The fix is at:
>
> https://github.com/azhou-nicira/ovs-review/tree/ovsdb-replication-sm-v3
>
> Would you please let me know if this version works any better?. Thanks!
>

​Sure. I would test and let you know.

Thanks
Numan
​
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [replication SMv2 7/7] ovsdb: Replication usability improvements

[Andy Zhou]

On Tue, Aug 30, 2016 at 4:17 AM, Numan Siddique  wrote:

>
>
> On Tue, Aug 30, 2016 at 1:11 AM, Andy Zhou  wrote:
>
>>
>>
>> On Mon, Aug 29, 2016 at 3:14 AM, Numan Siddique 
>> wrote:
>>
>>>
>>>
>>> On Sat, Aug 27, 2016 at 4:45 AM, Andy Zhou  wrote:
>>>
 Added the '--no-sync' option base on feedbacks of current
 implementation.

 Added appctl command "ovsdb-server/sync-status" based on feedbacks
 of current implementation.

 Added a test to simulate the integration of HA manager with OVSDB
 server using replication.

 Other documentation and API improvements.

 Signed-off-by: Andy Zhou 
 --

 I hope to get some review comments on the command line and appctl
 interfaces for replication. Since 2.6 is the first release of those
 interfaces, it is easier to making changes, compare to future
 releases.

 
 v1->v2: Fix creashes reported at:
 http://openvswitch.org/pipermail/dev/2016-August/078591.html
 ---

>>>
>>> ​I haven't tested these patches yet. This patch seems to have a white
>>> space warning when applied.
>>>
>> Thanks for the reported. I will fold the fix in the next version when
>> posting.
>>
>> In case it helps, you can also access the patches from my private repo at:
>>   https://github.com/azhou-nicira/ovs-review/tree/ovsdb-replic
>> ation-sm-v2
>>
>>
> ​
> Hi Andy,
> ​
> I am seeing the below crash when
>
> ​  - The ovsdb-server changes from
> ​master to standby and the active-ovsdb-server it is about to connect to
> is killed just before that or it is not reachable.
>
> ​  -
> ​The pacemaker OCF script calls the sync-status cmd soon after that.
>
>
> ​Please let me know if you need more information.​
>
>
> ​Core was generated by `ovsdb-server -vdbg 
> --log-file=/opt/stack/logs/ovsdb-server-sb.log
> --remote=puni'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x0041241d in replication_status () at ovsdb/replication.c:875
> 875SHASH_FOR_EACH (node, replication_dbs) {
> Missing separate debuginfos, use: dnf debuginfo-install
> glibc-2.23.1-10.fc24.x86_64 openssl-libs-1.0.2h-3.fc24.x86_64
> (gdb) bt
> #0  0x0041241d in replication_status () at ovsdb/replication.c:875
> #1  0x00406eda in ovsdb_server_get_sync_status (conn=0x1421fd0,
> argc=, argv=, config_=)
> at ovsdb/ovsdb-server.c:1480
> #2  0x004324ee in process_command (request=0x1421f30,
> conn=0x1421fd0) at lib/unixctl.c:313
> #3  run_connection (conn=0x1421fd0) at lib/unixctl.c:347
> #4  unixctl_server_run (server=server@entry=0x141e140) at
> lib/unixctl.c:400
> #5  0x00405bdc in main_loop (is_backup=0x7fff08062256,
> exiting=0x7fff08062257, run_process=0x0, remotes=0x7fff080622a0,
> unixctl=0x141e140,
> all_dbs=0x7fff080622e0, jsonrpc=0x13f6f00) at ovsdb/ovsdb-server.c:182
> #6  main (argc=, argv=) at
> ovsdb/ovsdb-server.c:430​
>
> Numan, thanks for the report. I think I spotted the bug:

Currently, when replication state machine is reset,  the state update takes
place after a round of main loop run. this time lag
could lead to the back trace in case the unixctl commands was issued during
this time lag.  I have a fix that add another
state to represent the reset condition.  The fix is at:

https://github.com/azhou-nicira/ovs-review/tree/ovsdb-replication-sm-v3

Would you please let me know if this version works any better?. Thanks!
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [replication SMv2 7/7] ovsdb: Replication usability improvements

[Numan Siddique]

On Tue, Aug 30, 2016 at 1:11 AM, Andy Zhou  wrote:

>
>
> On Mon, Aug 29, 2016 at 3:14 AM, Numan Siddique 
> wrote:
>
>>
>>
>> On Sat, Aug 27, 2016 at 4:45 AM, Andy Zhou  wrote:
>>
>>> Added the '--no-sync' option base on feedbacks of current
>>> implementation.
>>>
>>> Added appctl command "ovsdb-server/sync-status" based on feedbacks
>>> of current implementation.
>>>
>>> Added a test to simulate the integration of HA manager with OVSDB
>>> server using replication.
>>>
>>> Other documentation and API improvements.
>>>
>>> Signed-off-by: Andy Zhou 
>>> --
>>>
>>> I hope to get some review comments on the command line and appctl
>>> interfaces for replication. Since 2.6 is the first release of those
>>> interfaces, it is easier to making changes, compare to future
>>> releases.
>>>
>>> 
>>> v1->v2: Fix creashes reported at:
>>> http://openvswitch.org/pipermail/dev/2016-August/078591.html
>>> ---
>>>
>>
>> ​I haven't tested these patches yet. This patch seems to have a white
>> space warning when applied.
>>
> Thanks for the reported. I will fold the fix in the next version when
> posting.
>
> In case it helps, you can also access the patches from my private repo at:
>   https://github.com/azhou-nicira/ovs-review/tree/ovsdb-
> replication-sm-v2
>
>
​
Hi Andy,
​
I am seeing the below crash when

​  - The ovsdb-server changes from
​master to standby and the active-ovsdb-server it is about to connect to is
killed just before that or it is not reachable.

​  -
​The pacemaker OCF script calls the sync-status cmd soon after that.


​Please let me know if you need more information.​


​Core was generated by `ovsdb-server -vdbg
--log-file=/opt/stack/logs/ovsdb-server-sb.log --remote=puni'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0041241d in replication_status () at ovsdb/replication.c:875
875SHASH_FOR_EACH (node, replication_dbs) {
Missing separate debuginfos, use: dnf debuginfo-install
glibc-2.23.1-10.fc24.x86_64 openssl-libs-1.0.2h-3.fc24.x86_64
(gdb) bt
#0  0x0041241d in replication_status () at ovsdb/replication.c:875
#1  0x00406eda in ovsdb_server_get_sync_status (conn=0x1421fd0,
argc=, argv=, config_=)
at ovsdb/ovsdb-server.c:1480
#2  0x004324ee in process_command (request=0x1421f30,
conn=0x1421fd0) at lib/unixctl.c:313
#3  run_connection (conn=0x1421fd0) at lib/unixctl.c:347
#4  unixctl_server_run (server=server@entry=0x141e140) at lib/unixctl.c:400
#5  0x00405bdc in main_loop (is_backup=0x7fff08062256,
exiting=0x7fff08062257, run_process=0x0, remotes=0x7fff080622a0,
unixctl=0x141e140,
all_dbs=0x7fff080622e0, jsonrpc=0x13f6f00) at ovsdb/ovsdb-server.c:182
#6  main (argc=, argv=) at
ovsdb/ovsdb-server.c:430​


​Thanks
Numan
​
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [replication SMv2 7/7] ovsdb: Replication usability improvements

[Andy Zhou]

On Mon, Aug 29, 2016 at 3:14 AM, Numan Siddique  wrote:

>
>
> On Sat, Aug 27, 2016 at 4:45 AM, Andy Zhou  wrote:
>
>> Added the '--no-sync' option base on feedbacks of current
>> implementation.
>>
>> Added appctl command "ovsdb-server/sync-status" based on feedbacks
>> of current implementation.
>>
>> Added a test to simulate the integration of HA manager with OVSDB
>> server using replication.
>>
>> Other documentation and API improvements.
>>
>> Signed-off-by: Andy Zhou 
>> --
>>
>> I hope to get some review comments on the command line and appctl
>> interfaces for replication. Since 2.6 is the first release of those
>> interfaces, it is easier to making changes, compare to future
>> releases.
>>
>> 
>> v1->v2: Fix creashes reported at:
>> http://openvswitch.org/pipermail/dev/2016-August/078591.html
>> ---
>>
>
> ​I haven't tested these patches yet. This patch seems to have a white
> space warning when applied.
>
Thanks for the reported. I will fold the fix in the next version when
posting.

In case it helps, you can also access the patches from my private repo at:

https://github.com/azhou-nicira/ovs-review/tree/ovsdb-replication-sm-v2
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [replication SMv2 7/7] ovsdb: Replication usability improvements

[Andy Zhou]

Added the '--no-sync' option base on feedbacks of current
implementation.

Added appctl command "ovsdb-server/sync-status" based on feedbacks
of current implementation.

Added a test to simulate the integration of HA manager with OVSDB
server using replication.

Other documentation and API improvements.

Signed-off-by: Andy Zhou 
--

I hope to get some review comments on the command line and appctl
interfaces for replication. Since 2.6 is the first release of those
interfaces, it is easier to making changes, compare to future
releases.


v1->v2: Fix creashes reported at:
http://openvswitch.org/pipermail/dev/2016-August/078591.html
---
 Documentation/OVSDB-replication.md |   2 +-
 ovsdb/jsonrpc-server.c |   6 +
 ovsdb/jsonrpc-server.h |   1 +
 ovsdb/ovsdb-server.1.in|  24 ++--
 ovsdb/ovsdb-server.c   | 252 ++---
 ovsdb/replication.c|  80 +---
 ovsdb/replication.h|  28 -
 ovsdb/replication.man  |  16 ++-
 tests/ovsdb-server.at  | 105 ++--
 9 files changed, 401 insertions(+), 113 deletions(-)

diff --git a/Documentation/OVSDB-replication.md 
b/Documentation/OVSDB-replication.md
index 74c9500..a9ab5cc 100644
--- a/Documentation/OVSDB-replication.md
+++ b/Documentation/OVSDB-replication.md
@@ -144,7 +144,7 @@ commands are the following.
 between the active server and frees the memory used for the replication
 configuration.
 
--   ovsdb-server/set-sync-excluded-tables {db:table,...}: sets the tables list
+-   ovsdb-server/set-sync-exclude-tables {db:table,...}: sets the tables list
 that will be excluded from being replicated.
 
 -   ovsdb-server/get-sync-excluded-tables: gets the tables list that is
diff --git a/ovsdb/jsonrpc-server.c b/ovsdb/jsonrpc-server.c
index 45975a3..427026d 100644
--- a/ovsdb/jsonrpc-server.c
+++ b/ovsdb/jsonrpc-server.c
@@ -343,6 +343,12 @@ ovsdb_jsonrpc_server_reconnect(struct ovsdb_jsonrpc_server 
*svr, bool read_only)
 }
 }
 
+bool
+ovsdb_jsonrpc_server_is_read_only(struct ovsdb_jsonrpc_server *svr)
+{
+return svr->read_only;
+}
+
 void
 ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr)
 {
diff --git a/ovsdb/jsonrpc-server.h b/ovsdb/jsonrpc-server.h
index 955bbe4..f04b1e9 100644
--- a/ovsdb/jsonrpc-server.h
+++ b/ovsdb/jsonrpc-server.h
@@ -65,6 +65,7 @@ void ovsdb_jsonrpc_server_reconnect(struct 
ovsdb_jsonrpc_server *, bool read_onl
 
 void ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *);
 void ovsdb_jsonrpc_server_wait(struct ovsdb_jsonrpc_server *);
+bool ovsdb_jsonrpc_server_is_read_only(struct ovsdb_jsonrpc_server *);
 
 void ovsdb_jsonrpc_server_get_memory_usage(const struct ovsdb_jsonrpc_server *,
struct simap *usage);
diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
index d1ba83b..e2e96ae 100644
--- a/ovsdb/ovsdb-server.1.in
+++ b/ovsdb/ovsdb-server.1.in
@@ -36,7 +36,7 @@ If none is specified, the default is \fB@DBDIR@/conf.db\fR.  
The database
 files must already have been created and initialized using, for
 example, \fBovsdb\-tool create\fR.
 .
-.SH "ACTIVE and BACKUP "
+.SH "ACTIVE and BACKUP"
 \fBovsdb\-server\fR runs either as a backup server, or as an active server.
 When  \fBovsdb\-server\fR is running as a backup server, all transactions that
 can modify the database content, including the lock commands are rejected.
@@ -45,12 +45,12 @@ When \fBovsdb\-server\fR role changes, all existing client 
connection are
 reset, requiring clients to reconnect to the server.
 .PP
 By default, \fBovsdb\-server\fR runs as an active server, except when the
-\fB\-\-sync\-from=\fIserver\fR command line option is specified.  During
-runtime, \fBovsdb\-server\fR role can be switch by using appctl commands.
+\fB\-\-sync\-from=\fIserver\fR command line option is specified and without
+the \fB\-\-no\-sync option\fR.  During runtime, \fBovsdb\-server\fR role can 
be switch by using appctl commands.
 .PP
 \fBovsdb-server/connect\-active\-ovsdb\-server\fR switches
-\fBovsdb\-server\fR role into a backup server, Conversely,
-\fBovsdb-server/disconnect\-active\-ovsdb\-server\fR changes server into
+\fBovsdb\-server\fR into a backup server, Conversely,
+\fBovsdb-server/disconnect\-active\-ovsdb\-server\fR switches server into
 an active one.
 .
 .SH OPTIONS
@@ -220,12 +220,22 @@ specified by 
\fBovsdb\-server/set\-active\-ovsdb\-server\fR.
 .IP "\fBovsdb\-server/disconnect\-active\-ovsdb\-server"
 Causes \fBovsdb\-server\fR to  stop  synchronizing  its  databases with a 
active server.
 .
-.IP "\fBovsdb\-server/set\-sync\-excluded\-tables 
\fIdb\fB:\fItable\fR[\fB,\fIdb\fB:\fItable\fR]..."
+.IP "\fBovsdb\-server/set\-sync\-exclude\-tables 
\fIdb\fB:\fItable\fR[\fB,\fIdb\fB:\fItable\fR]..."
 Sets the \fItable\fR whitin \fIdb\fR that will be excluded from 
synchronization.
 .
-.IP