Oh, I see, from reading further in the thread, that this was indeed a misunderstanding. Well, in any case that new option to ovs-sandbox can be useful.
On Wed, Apr 04, 2018 at 04:00:20PM -0700, Ben Pfaff wrote: > I would like to support cluster-wide locks. They require extra work and > they require new OVSDB JSON-RPC protocol design (because locks are > currently per-server, not per-database). I do not currently have a > schedule for designing and implementing them. > > However, I am surprised that this is an issue for northd. For a > clustered database, ovn-northd always connects to the cluster leader. > There is at most one leader in the cluster at a given time, so as long > as ovn-northd obtains a lock on the leader, this should ensure that only > one ovn-northd is active at a time. There could be brief races, in > which two ovn-northds believe that they have the lock, but they should > not persist. > > You see different behavior, so there is a bug or a misunderstanding. > I don't see the same misbehavior, though, when I do a similar test in > the sandbox. If you apply the patches I just posted: > https://patchwork.ozlabs.org/patch/895184/ > https://patchwork.ozlabs.org/patch/895185/ > then you can try it out with: > make sandbox SANDBOXFLAGS='--ovn --sbdb-model=clustered --n-northds=3' > > On Wed, Mar 21, 2018 at 01:12:48PM -0700, aginwala wrote: > > :) The only thing is while using pacemaker, if the node that pacemaker if > > pointing to is down, all the active/standby northd nodes have to be updated > > to new node from the cluster. But will dig in more to see what else I can > > find. > > > > @Ben: Any suggestions further? > > > > > > Regards, > > > > On Wed, Mar 21, 2018 at 10:22 AM, Han Zhou <[email protected]> wrote: > > > > > > > > > > > On Wed, Mar 21, 2018 at 9:49 AM, aginwala <[email protected]> wrote: > > > > > >> Thanks Numan: > > >> > > >> Yup agree with the locking part. For now; yes I am running northd on one > > >> node. I might right a script to monitor northd in cluster so that if the > > >> node where it's running goes down, script can spin up northd on one other > > >> active nodes as a dirty hack. > > >> > > >> The "dirty hack" is pacemaker :) > > > > > > > > >> Sure, will await for the inputs from Ben too on this and see how complex > > >> would it be to roll out this feature. > > >> > > >> > > >> Regards, > > >> > > >> > > >> On Wed, Mar 21, 2018 at 5:43 AM, Numan Siddique <[email protected]> > > >> wrote: > > >> > > >>> Hi Aliasgar, > > >>> > > >>> ovsdb-server maintains locks per each connection and not across the db. > > >>> A workaround for you now would be to configure all the ovn-northd > > >>> instances > > >>> to connect to one ovsdb-server if you want to have active/standy. > > >>> > > >>> Probably Ben can answer if there is a plan to support ovsdb locks across > > >>> the db. We also need this support in networking-ovn as it also uses > > >>> ovsdb > > >>> locks. > > >>> > > >>> Thanks > > >>> Numan > > >>> > > >>> > > >>> On Wed, Mar 21, 2018 at 1:40 PM, aginwala <[email protected]> wrote: > > >>> > > >>>> Hi Numan: > > >>>> > > >>>> Just figured out that ovn-northd is running as active on all 3 nodes > > >>>> instead of one active instance as I continued to test further which > > >>>> results > > >>>> in db errors as per logs. > > >>>> > > >>>> > > >>>> # on node 3, I run ovn-nbctl ls-add ls2 ; it populates below logs in > > >>>> ovn-north > > >>>> 2018-03-21T06:01:59.442Z|00007|ovsdb_idl|WARN|transaction error: > > >>>> {"details":"Transaction causes multiple rows in \"Datapath_Binding\" > > >>>> table > > >>>> to have identical values (1) for index on column \"tunnel_key\". First > > >>>> row, with UUID 8c5d9342-2b90-4229-8ea1-001a733a915c, was inserted by > > >>>> this transaction. Second row, with UUID > > >>>> 8e06f919-4cc7-4ffc-9a79-20ce6663b683, > > >>>> existed in the database before this transaction and was not modified > > >>>> by the > > >>>> transaction.","error":"constraint violation"} > > >>>> > > >>>> In southbound datapath list, 2 duplicate records gets created for same > > >>>> switch. > > >>>> > > >>>> # ovn-sbctl list Datapath > > >>>> _uuid : b270ae30-3458-445f-95d2-b14e8ebddd01 > > >>>> external_ids : > > >>>> {logical-switch="4d6674e3-ff9f-4f38-b050-0fa9bec9e34d", > > >>>> name="ls2"} > > >>>> tunnel_key : 2 > > >>>> > > >>>> _uuid : 8e06f919-4cc7-4ffc-9a79-20ce6663b683 > > >>>> external_ids : > > >>>> {logical-switch="4d6674e3-ff9f-4f38-b050-0fa9bec9e34d", > > >>>> name="ls2"} > > >>>> tunnel_key : 1 > > >>>> > > >>>> > > >>>> > > >>>> # on nodes 1 and 2 where northd is running, it gives below error: > > >>>> 2018-03-21T06:01:59.437Z|00008|ovsdb_idl|WARN|transaction error: > > >>>> {"details":"cannot delete Datapath_Binding row > > >>>> 8e06f919-4cc7-4ffc-9a79-20ce6663b683 because of 17 remaining > > >>>> reference(s)","error":"referential integrity violation"} > > >>>> > > >>>> As per commit message, for northd I re-tried setting --ovnnb-db="tcp: > > >>>> 10.169.125.152:6641,tcp:10.169.125.131:6641,tcp:10.148.181.162:6641" > > >>>> and --ovnsb-db="tcp:10.169.125.152:6642,tcp:10.169.125.131:6642,tcp: > > >>>> 10.148.181.162:6642" and it did not help either. > > >>>> > > >>>> There is no issue if I keep running only one instance of northd on any > > >>>> of these 3 nodes. Hence, wanted to know is there something else > > >>>> missing here to make only one northd instance as active and rest as > > >>>> standby? > > >>>> > > >>>> > > >>>> Regards, > > >>>> > > >>>> On Thu, Mar 15, 2018 at 3:09 AM, Numan Siddique <[email protected]> > > >>>> wrote: > > >>>> > > >>>>> That's great > > >>>>> > > >>>>> Numan > > >>>>> > > >>>>> > > >>>>> On Thu, Mar 15, 2018 at 2:57 AM, aginwala <[email protected]> wrote: > > >>>>> > > >>>>>> Hi Numan: > > >>>>>> > > >>>>>> I tried on new nodes (kernel : 4.4.0-104-generic , Ubuntu 16.04)with > > >>>>>> fresh installation and it worked super fine for both sb and nb dbs. > > >>>>>> Seems > > >>>>>> like some kernel issue on the previous nodes when I re-installed > > >>>>>> raft patch > > >>>>>> as I was running different ovs version on those nodes before. > > >>>>>> > > >>>>>> > > >>>>>> For 2 HVs, I now set ovn-remote="tcp:10.169.125.152:6642, tcp: > > >>>>>> 10.169.125.131:6642, tcp:10.148.181.162:6642" and started > > >>>>>> controller and it works super fine. > > >>>>>> > > >>>>>> > > >>>>>> Did some failover testing by rebooting/killing the leader ( > > >>>>>> 10.169.125.152) and bringing it back up and it works as expected. > > >>>>>> Nothing weird noted so far. > > >>>>>> > > >>>>>> # check-cluster gives below data one of the node(10.148.181.162) post > > >>>>>> leader failure > > >>>>>> > > >>>>>> ovsdb-tool check-cluster /etc/openvswitch/ovnsb_db.db > > >>>>>> ovsdb-tool: leader /etc/openvswitch/ovnsb_db.db for term 2 has log > > >>>>>> entries only up to index 18446744073709551615, but index 9 was > > >>>>>> committed in > > >>>>>> a previous term (e.g. by /etc/openvswitch/ovnsb_db.db) > > >>>>>> > > >>>>>> > > >>>>>> For check-cluster, are we planning to add more output showing which > > >>>>>> node is active(leader), etc in upcoming versions ? > > >>>>>> > > >>>>>> > > >>>>>> Thanks a ton for helping sort this out. I think the patch looks good > > >>>>>> to be merged post addressing of the comments by Justin along with > > >>>>>> the man > > >>>>>> page details for ovsdb-tool. > > >>>>>> > > >>>>>> > > >>>>>> I will do some more crash testing for the cluster along with the > > >>>>>> scale test and keep you posted if something unexpected is noted. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Regards, > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Tue, Mar 13, 2018 at 11:07 PM, Numan Siddique <[email protected] > > >>>>>> > wrote: > > >>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> On Wed, Mar 14, 2018 at 7:51 AM, aginwala <[email protected]> wrote: > > >>>>>>> > > >>>>>>>> Sure. > > >>>>>>>> > > >>>>>>>> To add on , I also ran for nb db too using different port and > > >>>>>>>> Node2 crashes with same error : > > >>>>>>>> # Node 2 > > >>>>>>>> /usr/share/openvswitch/scripts/ovn-ctl --db-nb-addr=10.99.152.138 > > >>>>>>>> --db-nb-port=6641 --db-nb-cluster-remote-addr="tcp: > > >>>>>>>> 10.99.152.148:6645" --db-nb-cluster-local-addr="tcp: > > >>>>>>>> 10.99.152.138:6645" start_nb_ovsdb > > >>>>>>>> ovsdb-server: ovsdb error: /etc/openvswitch/ovnnb_db.db: cannot > > >>>>>>>> identify file type > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>> Hi Aliasgar, > > >>>>>>> > > >>>>>>> It worked for me. Can you delete the old db files in > > >>>>>>> /etc/openvswitch/ and try running the commands again ? > > >>>>>>> > > >>>>>>> Below are the commands I ran in my setup. > > >>>>>>> > > >>>>>>> Node 1 > > >>>>>>> ------- > > >>>>>>> sudo /usr/share/openvswitch/scripts/ovn-ctl > > >>>>>>> --db-sb-addr=192.168.121.91 --db-sb-port=6642 > > >>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>> --db-sb-cluster-local-addr=tcp:192.168.121.91:6644 start_sb_ovsdb > > >>>>>>> > > >>>>>>> Node 2 > > >>>>>>> --------- > > >>>>>>> sudo /usr/share/openvswitch/scripts/ovn-ctl > > >>>>>>> --db-sb-addr=192.168.121.87 --db-sb-port=6642 > > >>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>> --db-sb-cluster-local-addr="tcp:192.168.121.87:6644" > > >>>>>>> --db-sb-cluster-remote-addr="tcp:192.168.121.91:6644" > > >>>>>>> start_sb_ovsdb > > >>>>>>> > > >>>>>>> Node 3 > > >>>>>>> --------- > > >>>>>>> sudo /usr/share/openvswitch/scripts/ovn-ctl > > >>>>>>> --db-sb-addr=192.168.121.78 --db-sb-port=6642 > > >>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>> --db-sb-cluster-local-addr="tcp:192.168.121.78:6644" > > >>>>>>> --db-sb-cluster-remote-addr="tcp:192.168.121.91:6644" > > >>>>>>> start_sb_ovsdb > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> Thanks > > >>>>>>> Numan > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> > > >>>>>>>> On Tue, Mar 13, 2018 at 9:40 AM, Numan Siddique < > > >>>>>>>> [email protected]> wrote: > > >>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Tue, Mar 13, 2018 at 9:46 PM, aginwala <[email protected]> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Thanks Numan for the response. > > >>>>>>>>>> > > >>>>>>>>>> There is no command start_cluster_sb_ovsdb in the source code > > >>>>>>>>>> too. Is that in a separate commit somewhere? Hence, I used > > >>>>>>>>>> start_sb_ovsdb > > >>>>>>>>>> which I think would not be a right choice? > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Sorry, I meant start_sb_ovsdb. Strange that it didn't work for > > >>>>>>>>> you. Let me try it out again and update this thread. > > >>>>>>>>> > > >>>>>>>>> Thanks > > >>>>>>>>> Numan > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> # Node1 came up as expected. > > >>>>>>>>>> ovn-ctl --db-sb-addr=10.99.152.148 --db-sb-port=6642 > > >>>>>>>>>> --db-sb-create-insecure-remote=yes --db-sb-cluster-local-addr="tc > > >>>>>>>>>> p:10.99.152.148:6644" start_sb_ovsdb. > > >>>>>>>>>> > > >>>>>>>>>> # verifying its a clustered db with ovsdb-tool db-local-address > > >>>>>>>>>> /etc/openvswitch/ovnsb_db.db > > >>>>>>>>>> tcp:10.99.152.148:6644 > > >>>>>>>>>> # ovn-sbctl show works fine and chassis are being populated > > >>>>>>>>>> correctly. > > >>>>>>>>>> > > >>>>>>>>>> #Node 2 fails with error: > > >>>>>>>>>> /usr/share/openvswitch/scripts/ovn-ctl > > >>>>>>>>>> --db-sb-addr=10.99.152.138 --db-sb-port=6642 > > >>>>>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>>>>> --db-sb-cluster-remote-addr="tcp:10.99.152.148:6644" > > >>>>>>>>>> --db-sb-cluster-local-addr="tcp:10.99.152.138:6644" > > >>>>>>>>>> start_sb_ovsdb > > >>>>>>>>>> ovsdb-server: ovsdb error: /etc/openvswitch/ovnsb_db.db: cannot > > >>>>>>>>>> identify file type > > >>>>>>>>>> > > >>>>>>>>>> # So i did start the sb db the usual way using start_ovsdb to > > >>>>>>>>>> just get the db file created and killed the sb pid and re-ran > > >>>>>>>>>> the command > > >>>>>>>>>> which gave actual error where it complains for join-cluster > > >>>>>>>>>> command that is > > >>>>>>>>>> being called internally > > >>>>>>>>>> /usr/share/openvswitch/scripts/ovn-ctl > > >>>>>>>>>> --db-sb-addr=10.99.152.138 --db-sb-port=6642 > > >>>>>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>>>>> --db-sb-cluster-remote-addr="tcp:10.99.152.148:6644" > > >>>>>>>>>> --db-sb-cluster-local-addr="tcp:10.99.152.138:6644" > > >>>>>>>>>> start_sb_ovsdb > > >>>>>>>>>> ovsdb-tool: /etc/openvswitch/ovnsb_db.db: not a clustered > > >>>>>>>>>> database > > >>>>>>>>>> * Backing up database to /etc/openvswitch/ovnsb_db.db.b > > >>>>>>>>>> ackup1.15.0-70426956 > > >>>>>>>>>> ovsdb-tool: 'join-cluster' command requires at least 4 arguments > > >>>>>>>>>> * Creating cluster database /etc/openvswitch/ovnsb_db.db from > > >>>>>>>>>> existing one > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> # based on above error I killed the sb db pid again and try to > > >>>>>>>>>> create a local cluster on node then re-ran the join operation > > >>>>>>>>>> as per the > > >>>>>>>>>> source code function. > > >>>>>>>>>> ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db > > >>>>>>>>>> OVN_Southbound tcp:10.99.152.138:6644 tcp:10.99.152.148:6644 > > >>>>>>>>>> which still complains > > >>>>>>>>>> ovsdb-tool: I/O error: /etc/openvswitch/ovnsb_db.db: create > > >>>>>>>>>> failed (File exists) > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> # Node 3: I did not try as I am assuming the same failure as node > > >>>>>>>>>> 2 > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Let me know may know further. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On Tue, Mar 13, 2018 at 3:08 AM, Numan Siddique < > > >>>>>>>>>> [email protected]> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Hi Aliasgar, > > >>>>>>>>>>> > > >>>>>>>>>>> On Tue, Mar 13, 2018 at 7:11 AM, aginwala <[email protected]> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> Hi Ben/Noman: > > >>>>>>>>>>>> > > >>>>>>>>>>>> I am trying to setup 3 node southbound db cluster using raft10 > > >>>>>>>>>>>> <https://patchwork.ozlabs.org/patch/854298/> in review. > > >>>>>>>>>>>> > > >>>>>>>>>>>> # Node 1 create-cluster > > >>>>>>>>>>>> ovsdb-tool create-cluster /etc/openvswitch/ovnsb_db.db > > >>>>>>>>>>>> /root/ovs-reviews/ovn/ovn-sb.ovsschema tcp:10.99.152.148:6642 > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> A different port is used for RAFT. So you have to choose another > > >>>>>>>>>>> port like 6644 for example. > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> # Node 2 > > >>>>>>>>>>>> ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db > > >>>>>>>>>>>> OVN_Southbound tcp:10.99.152.138:6642 tcp:10.99.152.148:6642 > > >>>>>>>>>>>> --cid > > >>>>>>>>>>>> 5dfcb678-bb1d-4377-b02d-a380edec2982 > > >>>>>>>>>>>> > > >>>>>>>>>>>> #Node 3 > > >>>>>>>>>>>> ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db > > >>>>>>>>>>>> OVN_Southbound tcp:10.99.152.101:6642 tcp:10.99.152.138:6642 > > >>>>>>>>>>>> tcp:10.99.152.148:6642 --cid 5dfcb678-bb1d-4377-b02d-a380ed > > >>>>>>>>>>>> ec2982 > > >>>>>>>>>>>> > > >>>>>>>>>>>> # ovn remote is set to all 3 nodes > > >>>>>>>>>>>> external_ids:ovn-remote="tcp:10.99.152.148:6642, tcp: > > >>>>>>>>>>>> 10.99.152.138:6642, tcp:10.99.152.101:6642" > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>>> # Starting sb db on node 1 using below command on node 1: > > >>>>>>>>>>>> > > >>>>>>>>>>>> ovsdb-server --detach --monitor -vconsole:off -vraft -vjsonrpc > > >>>>>>>>>>>> --log-file=/var/log/openvswitch/ovsdb-server-sb.log > > >>>>>>>>>>>> --pidfile=/var/run/openvswitch/ovnsb_db.pid > > >>>>>>>>>>>> --remote=db:OVN_Southbound,SB_Global,connections > > >>>>>>>>>>>> --unixctl=ovnsb_db.ctl > > >>>>>>>>>>>> --private-key=db:OVN_Southbound,SSL,private_key > > >>>>>>>>>>>> --certificate=db:OVN_Southbound,SSL,certificate > > >>>>>>>>>>>> --ca-cert=db:OVN_Southbound,SSL,ca_cert > > >>>>>>>>>>>> --ssl-protocols=db:OVN_Southbound,SSL,ssl_protocols > > >>>>>>>>>>>> --ssl-ciphers=db:OVN_Southbound,SSL,ssl_ciphers > > >>>>>>>>>>>> --remote=punix:/var/run/openvswitch/ovnsb_db.sock > > >>>>>>>>>>>> /etc/openvswitch/ovnsb_db.db > > >>>>>>>>>>>> > > >>>>>>>>>>>> # check-cluster is returning nothing > > >>>>>>>>>>>> ovsdb-tool check-cluster /etc/openvswitch/ovnsb_db.db > > >>>>>>>>>>>> > > >>>>>>>>>>>> # ovsdb-server-sb.log below shows the leader is elected with > > >>>>>>>>>>>> only one server and there are rbac related debug logs with rpc > > >>>>>>>>>>>> replies and > > >>>>>>>>>>>> empty params with no errors > > >>>>>>>>>>>> > > >>>>>>>>>>>> 2018-03-13T01:12:02Z|00002|raft|DBG|server 63d1 added to > > >>>>>>>>>>>> configuration > > >>>>>>>>>>>> 2018-03-13T01:12:02Z|00003|raft|INFO|term 6: starting election > > >>>>>>>>>>>> 2018-03-13T01:12:02Z|00004|raft|INFO|term 6: elected leader by > > >>>>>>>>>>>> 1+ of 1 servers > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> Now Starting the ovsdb-server on the other clusters fails > > >>>>>>>>>>>> saying > > >>>>>>>>>>>> ovsdb-server: ovsdb error: /etc/openvswitch/ovnsb_db.db: cannot > > >>>>>>>>>>>> identify file type > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> Also noticed that man ovsdb-tool is missing cluster details. > > >>>>>>>>>>>> Might want to address it in the same patch or different. > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> Please advise to what is missing here for running ovn-sbctl > > >>>>>>>>>>>> show as this command hangs. > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> I think you can use the ovn-ctl command "start_cluster_sb_ovsdb" > > >>>>>>>>>>> for your testing (atleast for now) > > >>>>>>>>>>> > > >>>>>>>>>>> For your setup, I think you can start the cluster as > > >>>>>>>>>>> > > >>>>>>>>>>> # Node 1 > > >>>>>>>>>>> ovn-ctl --db-sb-addr=10.99.152.148 --db-sb-port=6642 > > >>>>>>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>>>>>> --db-sb-cluster-local-addr="tcp:10.99.152.148:6644" > > >>>>>>>>>>> start_cluster_sb_ovsdb > > >>>>>>>>>>> > > >>>>>>>>>>> # Node 2 > > >>>>>>>>>>> ovn-ctl --db-sb-addr=10.99.152.138 --db-sb-port=6642 > > >>>>>>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>>>>>> --db-sb-cluster-local-addr="tcp:10.99.152.138:6644" > > >>>>>>>>>>> --db-sb-cluster-remote-addr="tcp:10.99.152.148:6644" > > >>>>>>>>>>> start_cluster_sb_ovsdb > > >>>>>>>>>>> > > >>>>>>>>>>> # Node 3 > > >>>>>>>>>>> ovn-ctl --db-sb-addr=10.99.152.101 --db-sb-port=6642 > > >>>>>>>>>>> --db-sb-create-insecure-remote=yes > > >>>>>>>>>>> --db-sb-cluster-local-addr="tcp:10.99.152.101:6644" > > >>>>>>>>>>> --db-sb-cluster-remote-addr="tcp:10.99.152.148:6644" start_c > > >>>>>>>>>>> luster_sb_ovsdb > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> Let me know how it goes. > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks > > >>>>>>>>>>> Numan > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> _______________________________________________ > > >>>>>>>>>>>> discuss mailing list > > >>>>>>>>>>>> [email protected] > > >>>>>>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > >> _______________________________________________ > > >> discuss mailing list > > >> [email protected] > > >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > >> > > >> > > > _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
