Hi Jozef, I have tested and verified the below patch as you suggested in IRC,
https://git.opendaylight.org/gerrit/#/c/51071/3 First I have verified in legacy netvirt with our changes, and found that all the cases were working fine. [stack@control devstack]$ sudo systemctl restart openvswitch [stack@control devstack]$ sudo systemctl restart openvswitch [stack@control devstack]$ sudo systemctl restart openvswitch [stack@control devstack]$ [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x0, duration=6.046s, table=0, n_packets=0, n_bytes=0, dl_type=0x88cc actions=CONTROLLER:65535 cookie=0x0, duration=6.037s, table=0, n_packets=0, n_bytes=0, priority=0 actions=goto_table:20 cookie=0x0, duration=6.030s, table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:30 cookie=0x0, duration=6.020s, table=30, n_packets=0, n_bytes=0, priority=0 actions=goto_table:31 cookie=0x0, duration=6.056s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,39),resubmit(,40) cookie=0x0, duration=6.012s, table=40, n_packets=0, n_bytes=0, priority=0 actions=goto_table:50 cookie=0x0, duration=6.006s, table=50, n_packets=0, n_bytes=0, priority=0 actions=goto_table:60 cookie=0x0, duration=5.993s, table=60, n_packets=0, n_bytes=0, priority=0 actions=goto_table:70 cookie=0x0, duration=5.985s, table=70, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80 cookie=0x0, duration=5.976s, table=80, n_packets=0, n_bytes=0, priority=0 actions=goto_table:90 cookie=0x0, duration=5.973s, table=90, n_packets=0, n_bytes=0, priority=0 actions=goto_table:100 cookie=0x0, duration=5.963s, table=100, n_packets=0, n_bytes=0, priority=0 actions=goto_table:105 cookie=0x0, duration=5.959s, table=105, n_packets=0, n_bytes=0, priority=0 actions=goto_table:110 cookie=0x0, duration=5.952s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop [stack@control devstack]$ [stack@control devstack]$ [stack@control devstack]$ sudo systemctl stop openvswitch [stack@control devstack]$ sudo systemctl start openvswitch [stack@control devstack]$ [stack@control devstack]$ [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x0, duration=2.098s, table=0, n_packets=0, n_bytes=0, dl_type=0x88cc actions=CONTROLLER:65535 cookie=0x0, duration=2.091s, table=0, n_packets=0, n_bytes=0, priority=0 actions=goto_table:20 cookie=0x0, duration=2.083s, table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:30 cookie=0x0, duration=2.071s, table=30, n_packets=0, n_bytes=0, priority=0 actions=goto_table:31 cookie=0x0, duration=2.113s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,39),resubmit(,40) cookie=0x0, duration=2.063s, table=40, n_packets=0, n_bytes=0, priority=0 actions=goto_table:50 cookie=0x0, duration=2.056s, table=50, n_packets=0, n_bytes=0, priority=0 actions=goto_table:60 cookie=0x0, duration=2.041s, table=60, n_packets=0, n_bytes=0, priority=0 actions=goto_table:70 cookie=0x0, duration=2.036s, table=70, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80 cookie=0x0, duration=2.025s, table=80, n_packets=0, n_bytes=0, priority=0 actions=goto_table:90 cookie=0x0, duration=2.018s, table=90, n_packets=0, n_bytes=0, priority=0 actions=goto_table:100 cookie=0x0, duration=2.005s, table=100, n_packets=0, n_bytes=0, priority=0 actions=goto_table:105 cookie=0x0, duration=1.999s, table=105, n_packets=0, n_bytes=0, priority=0 actions=goto_table:110 cookie=0x0, duration=1.989s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop Second, the same changes have verified in new netvirt, and found that is not working. [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x0, duration=823.556s, table=0, n_packets=0, n_bytes=0, dl_type=0x88cc actions=CONTROLLER:65535 cookie=0x0, duration=823.550s, table=0, n_packets=0, n_bytes=0, priority=0 actions=goto_table:20 cookie=0x8000000, duration=111.691s, table=17, n_packets=0, n_bytes=0, priority=0,metadata=0x6000000000000000/0xf000000000000000 actions=write_metadata:0x7000000000000000/0xf000000000000000,goto_table:80 cookie=0x6800000, duration=111.222s, table=18, n_packets=0, n_bytes=0, priority=0 actions=goto_table:38 cookie=0x1080000, duration=17.410s, table=19, n_packets=0, n_bytes=0, priority=100,arp,arp_op=1 actions=group:5000 cookie=0x1080000, duration=17.410s, table=19, n_packets=0, n_bytes=0, priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resubmit(,17) cookie=0x1080000, duration=17.410s, table=19, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17) cookie=0x1030000, duration=17.410s, table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80 cookie=0x8000004, duration=17.410s, table=22, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535 cookie=0x0, duration=823.538s, table=30, n_packets=0, n_bytes=0, priority=0 actions=goto_table:31 cookie=0x0, duration=823.570s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,39),resubmit(,40) cookie=0x6900000, duration=111.837s, table=40, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x6900000, duration=111.836s, table=41, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x4000000, duration=111.837s, table=45, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17) cookie=0x8500000, duration=111.685s, table=48, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,49),resubmit(,50) cookie=0x8050001, duration=111.670s, table=50, n_packets=0, n_bytes=0, priority=10,reg4=0x1 actions=goto_table:51 cookie=0x8050000, duration=111.688s, table=50, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535,learn(table=49,hard_timeout=10,priority=0,cookie=0x8600000,NXM_OF_ETH_SRC[],load:0x1->NXM_NX_REG4[0..7]),goto_table:51 cookie=0x8030000, duration=111.667s, table=51, n_packets=0, n_bytes=0, priority=0 actions=goto_table:52 cookie=0x6800000, duration=111.222s, table=60, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17) cookie=0x0, duration=823.507s, table=70, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80 cookie=0x1030000, duration=17.410s, table=80, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17) cookie=0x8220000, duration=17.410s, table=81, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=823.500s, table=90, n_packets=0, n_bytes=0, priority=0 actions=goto_table:100 cookie=0x0, duration=823.487s, table=100, n_packets=0, n_bytes=0, priority=0 actions=goto_table:105 cookie=0x0, duration=823.480s, table=105, n_packets=0, n_bytes=0, priority=0 actions=goto_table:110 cookie=0x0, duration=823.472s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x6900000, duration=111.904s, table=251, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x6900000, duration=111.837s, table=252, n_packets=0, n_bytes=0, priority=0 actions=drop [stack@control devstack]$ [stack@control devstack]$ [stack@control devstack]$ [stack@control devstack]$ sudo systemctl stop openvswitch [stack@control devstack]$ sudo systemctl start openvswitch [stack@control devstack]$ [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int OFPST_FLOW reply (OF1.3) (xid=0x2): [stack@control devstack]$ [stack@control devstack]$ [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int OFPST_FLOW reply (OF1.3) (xid=0x2): For your reference attached the log file. Regards, Vinoth -----Original Message----- From: Jozef Bacigál [mailto:[email protected]] Sent: 02 February 2017 14:00 To: Bertrand Low <[email protected]>; Luis Gomez <[email protected]>; Jamo Luhrsen <[email protected]>; Abhijit Kumbhare <[email protected]> Cc: M Vinoth <[email protected]>; [email protected] Subject: RE: [openflowplugin-dev] handling of OVS restart +Abhijit Hi Bertrand, It is question for community if we wish to make cherry pick this patch into boron. Let's discuss on today's meeting. Jozef -----Original Message----- From: Bertrand Low [mailto:[email protected]] Sent: Thursday, February 2, 2017 5:20 AM To: Jozef Bacigál <[email protected]>; Luis Gomez <[email protected]>; Jamo Luhrsen <[email protected]> Cc: M Vinoth <[email protected]>; [email protected] Subject: RE: [openflowplugin-dev] handling of OVS restart Hi Jozef, I've ported your changes below to my local stable/boron repo: - All the patches related to 6802. - https://git.opendaylight.org/gerrit/#/c/51071/ With your patches, Bug 7462 (using stable/boron Legacy Netvirt) is no longer reproduced - the flows are installed correctly in the OVS restart scenario. Thanks. Do you plan to merge the changes over to the stable/boron branch as well? If not, I can help with that. Thanks, Bertrand -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Bertrand Low Sent: January 27, 2017 2:31 PM To: Luis Gomez <[email protected]>; Jamo Luhrsen <[email protected]> Cc: M Vinoth <[email protected]>; [email protected] Subject: Re: [openflowplugin-dev] handling of OVS restart Thanks Jozef for the 6802 patch collection, we are testing with that for the 7462 scenario - will let you know the results. Adding a minor delay between 'systemctl stop openvswitch' and 'systemctl start openvswitch' helps, but 'systemctl restart openvswitch' by default does not apply a sufficient delay between stopping and starting the process so it would be good if OF also handles the no-delay use case. Thanks, Bertrand -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Luis Gomez Sent: January 27, 2017 10:52 AM To: Jamo Luhrsen <[email protected]> Cc: M Vinoth <[email protected]>; [email protected] Subject: Re: [openflowplugin-dev] handling of OVS restart It is critical because only workaround is not acceptable :) > On Jan 27, 2017, at 10:45 AM, Jamo Luhrsen <[email protected]> wrote: > > > > On 01/26/2017 11:08 PM, Anil Vishnoi wrote: >> Bertrand, >> >> Yes the reported issue is same as you are facing with the NetVirt. >> It's happening because of the OVS don't reset/disconnect the >> connection properly (Don't send FIN/RST packet ) and netty assume >> that connection is just idle and not disconnected, so it does not >> disconnect the connection. Meanwhile if device tries to connect back, >> controller ignores that connection, because it thinks that device is already >> connection. Simple workaround is to add some minor delay (sleep .5), that >> will help with this situation. > > we can add the delays in our controlled environments to be nice, but > we can't always count on that in the wild. The workaround in the BZ > that Luis gave is to restart the controller. Seems like a drastic > workaround, imo. The bug is marked critical though, so hopefully something > can be done on our side to protect this. > > Thanks, > JamO > > >> Anil >> >> On Thu, Jan 26, 2017 at 9:35 PM, Luis Gomez <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Bertrand, >> >> The issue I see is reproducible with just 1 controller node and it is >> similar in the sense that to it requires an OF >> channel block (e.g. as a result of an OF agent reboot) followed by an OF >> channel restore. >> >> https://bugs.opendaylight.org/show_bug.cgi?id=7689 >> <https://bugs.opendaylight.org/show_bug.cgi?id=7689> >> >> BR/Luis >> >> >>> On Jan 26, 2017, at 12:39 PM, Bertrand Low <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi Luis and Jozef,____ >>> __ __ >>> In our ODL cluster setup, if the device issues a “systemctl restart >>> openvswitch”, or “systemctl stop openvswitch; >>> systemctl start openvswitch” (no delay between stopping and starting), >>> we’ve observed the issue of the br-int >>> controller being unable to reconnect to the OF master node. The OF >>> master node’s role will be “other” on the device and >>> it will keep toggling between “CONNECTING” and “BACKOFF” states. The >>> other OF nodes’ roles remains as “slave”. This >>> inability to reconnect to the OF master node (and the other nodes >>> remaining as slaves and not assuming mastership) >>> appears to persist until ODL is restarted.____ >>> Luis, is this the “switch hanging” symptom you’ve mentioned?____ >>> __ __ >>> In the scenario I’ve described, we have some logs up on bug 7462 >>> (https://bugs.opendaylight.org/show_bug.cgi?id=7462 >>> <https://bugs.opendaylight.org/show_bug.cgi?id=7462>) with DEBUG enabled >>> for org.opendaylight.openflowplugin. This was >>> observed while testing legacy netvirt; we will work on providing updated >>> logs for new netvirt as well.____ >>> __ __ >>> Thanks,____ >>> Bertrand____ >>> __ __ >>> *From:* [email protected] >>> <mailto:[email protected]> >>> [mailto:[email protected] >>> <mailto:[email protected]>] *On Behalf >>> Of *Luis Gomez >>> *Sent:* January 26, 2017 9:55 AM >>> *To:* Jozef Bacigál <[email protected] >>> <mailto:[email protected]>> >>> *Cc:* M Vinoth <[email protected] <mailto:[email protected]>>; >>> [email protected] >>> <mailto:[email protected]> >>> *Subject:* Re: [openflowplugin-dev] handling of OVS restart____ >>> __ __ >>> Jozef and all,____ >>> __ __ >>> FYI I have seen sporadic issues of switch hanging in controller after >>> switch connection is not properly closed, I am >>> currently working to see if I can consistently reproduce these issue in >>> the ODL CI.____ >>> __ __ >>> BR/Luis____ >>> __ __ >>> __ __ >>> >>> On Jan 26, 2017, at 12:23 AM, Jozef Bacigál >>> <[email protected] <mailto:[email protected]>> >>> wrote:____ >>> __ __ >>> May I ask you to run this non-working case again but with ____ >>> __ __ >>> org.opendaylight.openflowplugin.impl DEBUG____ >>> __ __ >>> and send me the logs please ?____ >>> __ __ >>> Thanks____ >>> __ __ >>> Jozef____ >>> __ __ >>> __ __ >>> >>> ----------------------------------------------------------------------------------------------------------------------------- >>> *Od:* M Vinoth <[email protected] <mailto:[email protected]>> >>> *Odoslané:* 25. januára 2017 15:31 >>> *Komu:* Jozef Bacigál >>> *Kópia:* [email protected] >>> <mailto:[email protected]> >>> *Predmet:* RE: handling of OVS restart____ >>> ____ >>> Hi Jozef,____ >>> ____ >>> As you suggested I have checkout the last patch of 6802. And now I >>> can see all the changes.____ >>> ____ >>> And I have tested this changes with new netvirt build.____ >>> ____ >>> But still the dump flows are not installed properly. And other test >>> cases are working fine (1. Node disconnect 2. >>> Br-int deletion)____ >>> ____ >>> Please find my analysis in below for main difference between working >>> and failing cases,____ >>> ____ >>> *Working cases (attached in mail : try_1 document):*____ >>> ____ >>> 2017-01-25 21:14:49,043 | INFO | lt-dispatcher-17 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> Entity{type='ovsdb', >>> >>> id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b}]} >>> has no owner, cleaning up the operational data store____ >>> ____ >>> 2017-01-25 21:14:49,574 | INFO | DBConnNotifSer-5 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> InstanceIdentifier >>> KeyedInstanceIdentifier{targetType=interface >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, >>> >>> path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey >>> [_topologyId=Uri [_value=ovsdb:1]]], >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey >>> [_nodeId=Uri >>> [_value=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b]]]]} generated >>> for device connection >>> ConnectionInfo [Remote-address=10.106.138.164, Remote-port=32850, >>> Local-address10.106.138.251, Local-port=6640, >>> type=PASSIVE]____ >>> 2017-01-25 21:14:49,574 | INFO | DBConnNotifSer-5 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | OVSDB >>> entity Entity{type='ovsdb', >>> >>> id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b}]} >>> is registered for ownership.____ >>> 2017-01-25 21:14:49,692 | INFO | lt-dispatcher-17 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> handleOwnershipChanged: *this* southbound plugin instance >>> is an OWNER of the device ConnectionInfo >>> [Remote-address=10.106.138.164, Remote-port=32850, >>> Local-address10.106.138.251, Local-port=6640, type=PASSIVE]____ >>> ____ >>> *Failing Cases (attached in mail : try_2 document)*____ >>> ____ >>> 2017-01-25 21:15:38,751 | INFO | entLoopGroup-5-4 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | Library >>> disconnected PASSIVE from /10.106.138.164:32850 >>> <http://10.106.138.164:32850> to /10.106.138.251:6640 >>> <http://10.106.138.251:6640>. Cleaning up the operational >>> data store____ >>> ____ >>> 2017-01-25 21:15:39,283 | INFO | DBConnNotifSer-4 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> InstanceIdentifier >>> KeyedInstanceIdentifier{targetType=interface >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, >>> >>> path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey >>> [_topologyId=Uri [_value=ovsdb:1]]], >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey >>> [_nodeId=Uri >>> [_value=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b]]]]} generated >>> for device connection >>> ConnectionInfo [Remote-address=10.106.138.164, Remote-port=32862, >>> Local-address10.106.138.251, Local-port=6640, >>> type=PASSIVE]____ >>> 2017-01-25 21:15:39,284 | INFO | DBConnNotifSer-4 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | OVSDB >>> entity Entity{type='ovsdb', >>> >>> id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b}]} >>> is registered for ownership.____ >>> 2017-01-25 21:15:39,309 | INFO | lt-dispatcher-17 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> handleOwnershipChanged: no change in ownership for >>> ConnectionInfo [Remote-address=10.106.138.164, Remote-port=32862, >>> Local-address10.106.138.251, Local-port=6640, >>> type=PASSIVE]. Ownership status is : NON-OWNER____ >>> 2017-01-25 21:15:39,333 | INFO | lt-dispatcher-19 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> handleOwnershipChanged: *this* southbound plugin instance >>> is an OWNER of the device ConnectionInfo >>> [Remote-address=10.106.138.164, Remote-port=32862, >>> Local-address10.106.138.251, Local-port=6640, type=PASSIVE]____ >>> ____ >>> *Example:*____ >>> [stack@control devstack]$ sudo systemctl stop openvswitch____ >>> [stack@control devstack]$ sudo systemctl start openvswitch____ >>> [stack@control devstack]$____ >>> [stack@control devstack]$____ >>> [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows >>> br-int____ >>> OFPST_FLOW reply (OF1.3) (xid=0x2):____ >>> ____ >>> I will test more scenario’s and then I will let you know further.____ >>> ____ >>> Regards,____ >>> Vinoth____ >>> ____ >>> ____ >>> *From:* Jozef Bacigál [mailto:[email protected] >>> <mailto:[email protected]>] >>> *Sent:* 25 January 2017 15:53 >>> *To:* M Vinoth <[email protected] <mailto:[email protected]>> >>> *Cc:* [email protected] >>> <mailto:[email protected]> >>> *Subject:* Re: handling of OVS restart____ >>> ____ >>> Hi Vinoth,____ >>> ____ >>> yes the patch for bug 6082 should prevent this kind of issues. We >>> trying to make some test with stop/start >>> connection without delay on this patch. I'll let you know if we were >>> able to reproduce the issue. ____ >>> ____ >>> Anyway as I said this patch shall hold all information about >>> switches for few seconds/milliseconds (configurable) >>> and tha just replace broken connection with correct one without any >>> changes in behavior.____ >>> ____ >>> I will let you know.____ >>> ____ >>> Jozef____ >>> ____ >>> ____ >>> >>> ----------------------------------------------------------------------------------------------------------------------------- >>> *Od:* M Vinoth <[email protected] <mailto:[email protected]>> >>> *Odoslané:* 24. januára 2017 13:40 >>> *Komu:* Jozef Bacigál >>> *Kópia:* [email protected] >>> <mailto:[email protected]> >>> *Predmet:* FW: handling of OVS restart____ >>> ____ >>> Hi Jozef,____ >>> ____ >>> I have seen your patch for bug:6082., and observe that implemented >>> related to the below issues.____ >>> https://git.opendaylight.org/gerrit/#/q/topic:%22Bug+6802%22 >>> <https://git.opendaylight.org/gerrit/#/q/topic:%22Bug+6802%22>____ >>> ____ >>> The issue is when OVS stop and start immediately without delay, that >>> time br-int flows were not available. And node >>> also disconnect.____ >>> And I have notice that life cycle service and Device context service >>> is not properly invoked when failed cases.____ >>> In below mail I have highlighted comparatively with working as well >>> as failed cases.____ >>> Whether the below issue is addressed in your patch.____ >>> Please validate the below mail and share your inputs on this.____ >>> Thanks.____ >>> ____ >>> Regards,____ >>> Vinoth____ >>> *From:* [email protected] >>> <mailto:[email protected]> >>> [mailto:[email protected] >>> <mailto:[email protected]>] *On Behalf Of >>> *M Vinoth >>> *Sent:* 24 January 2017 15:33 >>> *To:* Bertrand Low <[email protected] >>> <mailto:[email protected]>>; Vishal Thapar <[email protected] >>> <mailto:[email protected]>>; >>> [email protected] >>> <mailto:[email protected]> >>> *Cc:* [email protected] >>> <mailto:[email protected]>; >>> [email protected] >>> <mailto:[email protected]> >>> *Subject:* Re: [netvirt-dev] handling of OVS restart____ >>> ____ >>> Hi Vishal,____ >>> ____ >>> I have tested the following behaviour in new netvirt in 3-node >>> cluster,____ >>> ____ >>> The following commands requested without delay,____ >>> [stack@control devstack]$ sudo systemctl stop openvswitch____ >>> [stack@control devstack]$ sudo systemctl start openvswitch____ >>> ____ >>> And found that one of the node is disconnected from 3-node >>> cluster.____ >>> ____ >>> [stack@control devstack]$ sudo ovs-vsctl show____ >>> 655c51fc-c86b-4cd7-bb0c-7c5d273b457b____ >>> Manager "tcp:10.106.138.251:6640 >>> <http://10.106.138.251:6640>"____ >>> is_connected: true____ >>> Manager "tcp:10.106.138.248:6640 >>> <http://10.106.138.248:6640>"____ >>> is_connected: true____ >>> Manager "tcp:10.106.138.244:6640 >>> <http://10.106.138.244:6640>"____ >>> is_connected: true____ >>> Bridge br-int____ >>> Controller "tcp:10.106.138.251:6653 >>> <http://10.106.138.251:6653>"____ >>> Controller "tcp:10.106.138.248:6653 >>> <http://10.106.138.248:6653>"____ >>> is_connected: true____ >>> Controller "tcp:10.106.138.244:6653 >>> <http://10.106.138.244:6653>"____ >>> is_connected: true____ >>> fail_mode: secure____ >>> Port br-int____ >>> Interface br-int____ >>> type: internal____ >>> ovs_version: "2.5.0"____ >>> ____ >>> Verified that the internal bridge default flows are not >>> available.____ >>> ____ >>> [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows >>> br-int____ >>> OFPST_FLOW reply (OF1.3) (xid=0x2):____ >>> [stack@control devstack]$____ >>> ____ >>> The same behaviour in legacy netvirt too, but additionally one more >>> cases were failed in legacy that br-int is not >>> available.____ >>> ____ >>> I will share the following execution flows based on the log which >>> is analysed in legacy netvirt.____ >>> *_Working cases execution flows,_*____ >>> *ß**------------------ ------ Stopping the service >>> ------------------------------------------**à** **(A)*____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59630 >>> <http://10.106.138.164:59630>, NodeId:openflow:35760623087526____ >>> OvsdbConnectionService - Connection closed ConnectionInfo____ >>> OvsdbConnectionManager - Library disconnected PASSIVE from >>> /10.106.138.164:57042 <http://10.106.138.164:57042> to >>> /10.106.138.251:6640 <http://10.106.138.251:6640>. Cleaning up the >>> operational data store____ >>> OvsdbConnectionManager - handleOwnershipChanged: *this* instance is >>> elected as an owner of the device____ >>> OvsdbConnectionManager - has no owner, cleaning up the operational >>> data store____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**à** **(B) *____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - processOvsdb Node Delete :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> *ß**------------------ ------ Setting Up the cluster service >>> ------------------------------------------**à** **(C)*____ >>> ConnectionAdapterImpl - Hello received / branch____ >>> DeviceManagerImpl - ConnectionEvent: Device connected to >>> controller,____ >>> SalRoleServiceImpl - SetRole called with input:SetRoleInput____ >>> SalRoleServiceImpl - Requesting state change to BECOMESLAVE____ >>> SalRoleServiceImpl - RoleChangeTask called on >>> device:openflow:35760623087526 OFPRole:BECOMESLAVE____ >>> RoleService - getGenerationIdFromDevice called for device: >>> openflow:35760623087526____ >>> LifecycleServiceImpl - Registered clustering MASTER services for >>> node openflow:35760623087526____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> LoggingHandler - RECEIVED: [id: 0x2cd517d7, L:/10.106.138.251:6640 >>> <http://10.106.138.251:6640> - >>> R:/10.106.138.164:57066 <http://10.106.138.164:57066>]____ >>> RoleService - submitRoleChange onSuccess for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> OvsdbConnectionManager - Library connected PASSIVE from >>> /10.106.138.164:57066 <http://10.106.138.164:57066> to >>> /10.106.138.251:6640 <http://10.106.138.251:6640>____ >>> OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57066 >>> <http://10.106.138.164:57066>____ >>> OvsdbConnectionManager - generated for device connection >>> ConnectionInfo____ >>> OvsdbConnectionManager - OVSDB entity Entity, is registered for >>> ownership.____ >>> LifecycleServiceImpl - Starting clustering MASTER services for node >>> openflow:35760623087526____ >>> DeviceContextImpl - Starting device context cluster services for >>> node openflow:35760623087526____ >>> OvsdbConnectionManager - handleOwnershipChanged: *this* southbound >>> plugin instance is an OWNER of the device >>> ConnectionInfo____ >>> OvsdbConnectionInstance - Monitoring database: Open_vSwitch____ >>> DeviceInitializationUtils - IP address of switch is: >>> /10.106.138.164:59640 <http://10.106.138.164:59640>____ >>> DeviceInitializationUtils - Static node Uri >>> [_value=openflow:35760623087526] info: OFPMPMETERFEATURES collected____ >>> StringValueObjectFactory - Instantiated factory for class >>> >>> org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.yang.types.rev130715.MacAddress____ >>> DeviceInitializationUtils - Static node Uri >>> [_value=openflow:35760623087526] info: OFPMPPORTDESC collected____ >>> SalRoleServiceImpl - SetRole called with input:____ >>> SalRoleServiceImpl - Requesting state change to BECOMEMASTER____ >>> RoleService - getGenerationIdFromDevice called for device: >>> openflow:35760623087526____ >>> StatisticsContextImpl - Starting statistics context cluster services >>> for node openflow:35760623087526____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMEMASTER____ >>> RoleService - submitRoleChange onSuccess for device:Uri >>> [_value=openflow:35760623087526], role:BECOMEMASTER____ >>> StatisticsManagerImpl - Scheduling statistics poll for device: Uri >>> [_value=openflow:35760623087526]____ >>> *<-------- Updating the internal bridge >>> ----------------------------**à** **(D)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Create :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> OF13Provider - initializeOFFlowRules: bridgeName: br-int____ >>> ControllerUpdateCommand - Register ODL controllers : >>> ControllerEntry____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> ____ >>> *_Failure cases execution flows (Without delay),_*____ >>> * *____ >>> *ß**------------------ ------ Stopping the service >>> ------------------------------------------**à** **(A)*____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59640 >>> <http://10.106.138.164:59640>, NodeId:openflow:35760623087526____ >>> OvsdbConnectionService - Connection closed ConnectionInfo____ >>> OvsdbConnectionManager - Library disconnected PASSIVE from >>> /10.106.138.164:57066 <http://10.106.138.164:57066> to >>> /10.106.138.251:6640 <http://10.106.138.251:6640>. Cleaning up the >>> operational data store____ >>> OvsdbConnectionManager - has no owner, cleaning up the operational >>> data store____ >>> *ß**------------------ ------ Setting Up the cluster service >>> ------------------------------------------**à** **(C)*____ >>> ConnectionAdapterImpl - Hello received / branch____ >>> DeviceManagerImpl - ConnectionEvent: Device connected to controller, >>> Device:/10.106.138.164:59658 >>> <http://10.106.138.164:59658>, NodeId:Uri >>> [_value=openflow:35760623087526]____ >>> SalRoleServiceImpl - SetRole called with input:SetRoleInput____ >>> SalRoleServiceImpl - Requesting state change to BECOMESLAVE____ >>> SalRoleServiceImpl - RoleChangeTask called on >>> device:openflow:35760623087526 OFPRole:BECOMESLAVE____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> RoleService - submitRoleChange onSuccess for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> LoggingHandler - RECEIVED: [id: 0x2610f6bd, L:/10.106.138.251:6640 >>> <http://10.106.138.251:6640> - >>> R:/10.106.138.164:57084 <http://10.106.138.164:57084>]____ >>> OvsdbConnectionManager - Library connected PASSIVE from >>> /10.106.138.164:57084 <http://10.106.138.164:57084> to >>> /10.106.138.251:6640 <http://10.106.138.251:6640>____ >>> OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57084 >>> <http://10.106.138.164:57084>____ >>> OvsdbConnectionManager - generated for device connection >>> ConnectionInfo____ >>> OvsdbConnectionManager - OVSDB entity, is registered for >>> ownership.____ >>> OvsdbConnectionManager - handleOwnershipChanged: *this* southbound >>> plugin instance is an OWNER of the device >>> ConnectionInfo____ >>> OvsdbConnectionInstance - Monitoring database: Open_vSwitch____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**à** **(B)*____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - processOvsdb Node Delete :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> ControllerUpdateCommand - Register ODL controllers : {} bridges >>> detail : {}____ >>> ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for >>> OVSDB Bridge : null____ >>> BridgeRemovedCommand - Bridge Deleted:____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59658 >>> <http://10.106.138.164:59658>, NodeId:openflow:35760623087526____ >>> OvsdbPortRemoveCommand - Bridge not found for port____ >>> *<-------- Updating the internal bridge >>> ----------------------------**à** **(D)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Create :____ >>> SouthboundImpl - addBridge: node:____ >>> SouthboundImpl - addBridge: result: true____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> OF13Provider - initializeOFFlowRules: bridgeName: br-int____ >>> BridgeUpdateCommand - Added ovsdb Bridge name: OvsdbBridgeName >>> [_value=br-int] uuid: null____ >>> ControllerUpdateCommand - Register ODL controllers : >>> ControllerEntry____ >>> ProtocolUpdateCommand - Updated ProtocolEntry : OpenFlow13 for OVSDB >>> Bridge : br-int____ >>> *ß**------------------ ------ Setting Up the cluster service >>> ------------------------------------------**à** **(C)*____ >>> ConnectionAdapterImpl - Hello received / branch____ >>> DeviceManagerImpl - ConnectionEvent: Device connected to controller, >>> Device:/10.106.138.164:59670 >>> <http://10.106.138.164:59670>, NodeId:Uri >>> [_value=openflow:35760623087526]____ >>> SalRoleServiceImpl - Requesting state change to BECOMESLAVE____ >>> SalRoleServiceImpl - RoleChangeTask called on >>> device:openflow:35760623087526 OFPRole:BECOMESLAVE____ >>> RoleService - getGenerationIdFromDevice called for device: >>> openflow:35760623087526____ >>> LifecycleServiceImpl - Registered clustering MASTER services for >>> node openflow:35760623087526____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> RoleService - submitRoleChange onSuccess for device:____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**à** **(B)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> *<-------- Updating the internal bridge >>> ----------------------------**à** **(D)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> OF13Provider - initializeOFFlowRules: bridgeName: br-int____ >>> ControllerUpdateCommand - Register ODL controllers : {} bridges >>> detail : {}____ >>> ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for >>> OVSDB Bridge : null____ >>> BridgeRemovedCommand - Bridge Deleted:____ >>> OvsdbPortRemoveCommand - Bridge not found for port Port :____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59670 >>> <http://10.106.138.164:59670>, NodeId:openflow:35760623087526____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**à** **(B)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> ____ >>> I have observe that the above working cases are properly executing >>> the life cycle service and device context for >>> cluster service, it means that properly maintain the context >>> information.____ >>> ____ >>> Please share your inputs.____ >>> ____ >>> Regards,____ >>> Vinoth____ >>> ____ >>> ____ >>> *From:* [email protected] >>> <mailto:[email protected]> >>> [mailto:[email protected] >>> <mailto:[email protected]>] *On Behalf Of >>> *Bertrand Low >>> *Sent:* 23 January 2017 11:02 >>> *To:* Vishal Thapar <[email protected] >>> <mailto:[email protected]>>; >>> [email protected] >>> <mailto:[email protected]> >>> *Subject:* Re: [netvirt-dev] handling of OVS restart____ >>> ____ >>> Thanks Vishal,____ >>> ____ >>> The bridge deletion upon disconnect does seem like a potential issue >>> in legacy netvirt. I’ll do some testing on >>> this.____ >>> ____ >>> Cheers,____ >>> Bertrand____ >>> ____ >>> *From:* Vishal Thapar [mailto:[email protected] >>> <mailto:[email protected]>] >>> *Sent:* January 22, 2017 7:59 PM >>> *To:* Bertrand Low <[email protected] >>> <mailto:[email protected]>>; [email protected] >>> <mailto:[email protected]> >>> *Subject:* RE: handling of OVS restart____ >>> ____ >>> Hi Bertrand,____ >>> ____ >>> Can you try with new netvirt? I am not that familiar with legacy >>> netvirt but new netvirt doesn’t have code to >>> delete bridge on disconnect. >>> https://bugs.opendaylight.org/show_bug.cgi?id=7488 >>> <https://bugs.opendaylight.org/show_bug.cgi?id=7488> fixed an issue >>> with dpnid changing which was impacting cluster >>> setups.____ >>> ____ >>> So, for now I’d say that this issue is not seen in new netvirt >>> unless proven otherwise J____ >>> ____ >>> Regards,____ >>> Vishal.____ >>> ____ >>> *From:* [email protected] >>> <mailto:[email protected]> >>> [mailto:[email protected] >>> <mailto:[email protected]>] *On Behalf Of >>> *Bertrand Low >>> *Sent:* 23 January 2017 08:36 >>> *To:* [email protected] >>> <mailto:[email protected]> >>> *Subject:* [netvirt-dev] handling of OVS restart____ >>> ____ >>> Hi all,____ >>> ____ >>> Has anyone run into issues with ODL handling OVS restart?____ >>> ____ >>> The scenario is like this:____ >>> ____ >>> 1) Connect device to an ODL cluster (i.e. at least 3), see that >>> br-int and the pipeline flows are installed >>> successfully____ >>> 2) Execute “restart openvswitch” on the device, or execute >>> “stop openvswitch; start openvswitch” (i.e. no >>> delay between starting and stopping openvswitch)____ >>> 3) Once the device reconnects to ODL, there are two failure >>> scenarios observed:____ >>> a. There is no br-int (Bug 7461)____ >>> 1. A connection flap is observed for br-int, sometimes with >>> br-int assigned a new dpnid____ >>> OR____ >>> b. br-int is installed, but without any pipeline flows (Bug >>> 7462)____ >>> ____ >>> Seems to me like the “restart openvswitch” use case should be >>> handled by ODL – please let me know if my >>> understanding is incorrect.____ >>> ____ >>> I’ve been testing with legacy netvirt but this timing issue may be >>> relevant to new netvirt as well.____ >>> If anyone has any insight on new netvirt’s handling of OVS restart, >>> please share. Thanks!____ >>> ____ >>> Bertrand____ >>> >>> >>> ::DISCLAIMER:: >>> >>> ----------------------------------------------------------------------------------------------------------------------------------------------------____ >>> The contents of this e-mail and any attachment(s) are confidential >>> and intended for the named recipient(s) only. >>> E-mail transmission is not guaranteed to be secure or error-free as >>> information could be intercepted, corrupted, >>> lost, destroyed, arrive late or incomplete, or may contain viruses >>> in transmission. The e mail and its contents >>> (with or without referred errors) shall therefore not attach any >>> liability on the originator or HCL or its affiliates. >>> Views or opinions, if any, presented in this email are solely those >>> of the author and may not necessarily reflect the >>> views or opinions of HCL or its affiliates. Any form of >>> reproduction, dissemination, copying, disclosure, >>> modification, >>> distribution and / or publication of this message without the prior >>> written consent of authorized representative of >>> HCL is strictly prohibited. If you have received this email in error >>> please delete it and notify the sender >>> immediately. >>> Before opening any email and/or attachments, please check them for >>> viruses and other defects.____ >>> >>> ----------------------------------------------------------------------------------------------------------------------------------------------------____ >>> ____ >>> Jozef*Bacigál*____ >>> Software Engineer____ >>> >>> Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia >>> R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia >>> +421 908 766 972 <tel:+421%20908%20766%20972> / >>> [email protected] <mailto:[email protected]> >>> reception: +421 2 206 65 114 / www.pantheon.tech >>> <http://www.pantheon.tech/>____ >>> logo____ >>> ____ >>> ____ >>> Jozef*Bacigál*____ >>> Software Engineer____ >>> >>> Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia >>> R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia >>> +421 908 766 972 <tel:+421%20908%20766%20972> / >>> [email protected] <mailto:[email protected]> >>> reception: +421 2 206 65 114 / www.pantheon.tech >>> <http://www.pantheon.tech/>____ >>> logo____ >>> ____ >>> _______________________________________________ >>> openflowplugin-dev mailing list >>> [email protected] >>> <mailto:[email protected]> >>> >>> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >>> >>> <https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev> >> >> >> _______________________________________________ >> openflowplugin-dev mailing list >> [email protected] >> <mailto:[email protected]> >> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >> >> <https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev> >> >> >> >> >> -- >> Thanks >> Anil >> >> >> _______________________________________________ >> openflowplugin-dev mailing list >> [email protected] >> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >> _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev JozefBacigál Software Engineer Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia +421 908 766 972 / [email protected] reception: +421 2 206 65 114 / www.pantheon.tech [logo] _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
