Hi, Bertrand
let me do one more patch set to update the inital times in cfg and yang files and then we can prepare cherry picks for boron. In master will be one more update wich I will prepare and test before the merge. Jozef ________________________________ Od: Bertrand Low <[email protected]> Odoslané: 3. februára 2017 4:27 Komu: Abhijit Kumbhare; Jozef Bacigál Kópia: Luis Gomez; Jamo Luhrsen; Abhijit Kumbhare; M Vinoth; [email protected] Predmet: RE: [openflowplugin-dev] handling of OVS restart Great, thanks Abhijit and Jozef. Jozef, please let me know if I can help with backporting the changes to Boron. I’m thinking the backport will happen once these patches have been merged into master. Thanks, Bertrand From: Abhijit Kumbhare [mailto:[email protected]] Sent: February 2, 2017 10:19 AM To: Jozef Bacigál <[email protected]> Cc: Bertrand Low <[email protected]>; Luis Gomez <[email protected]>; Jamo Luhrsen <[email protected]>; Abhijit Kumbhare <[email protected]>; M Vinoth <[email protected]>; [email protected] Subject: Re: [openflowplugin-dev] handling of OVS restart OK - as per the decision in the meeting today - it is OK to backport to Boron since this is a bug fix not a feature. On Thu, Feb 2, 2017 at 12:29 AM, Jozef Bacigál <[email protected]<mailto:[email protected]>> wrote: +Abhijit Hi Bertrand, It is question for community if we wish to make cherry pick this patch into boron. Let's discuss on today's meeting. Jozef -----Original Message----- From: Bertrand Low [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, February 2, 2017 5:20 AM To: Jozef Bacigál <[email protected]<mailto:[email protected]>>; Luis Gomez <[email protected]<mailto:[email protected]>>; Jamo Luhrsen <[email protected]<mailto:[email protected]>> Cc: M Vinoth <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: RE: [openflowplugin-dev] handling of OVS restart Hi Jozef, I've ported your changes below to my local stable/boron repo: - All the patches related to 6802. - https://git.opendaylight.org/gerrit/#/c/51071/ With your patches, Bug 7462 (using stable/boron Legacy Netvirt) is no longer reproduced - the flows are installed correctly in the OVS restart scenario. Thanks. Do you plan to merge the changes over to the stable/boron branch as well? If not, I can help with that. Thanks, Bertrand -----Original Message----- From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Bertrand Low Sent: January 27, 2017 2:31 PM To: Luis Gomez <[email protected]<mailto:[email protected]>>; Jamo Luhrsen <[email protected]<mailto:[email protected]>> Cc: M Vinoth <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [openflowplugin-dev] handling of OVS restart Thanks Jozef for the 6802 patch collection, we are testing with that for the 7462 scenario - will let you know the results. Adding a minor delay between 'systemctl stop openvswitch' and 'systemctl start openvswitch' helps, but 'systemctl restart openvswitch' by default does not apply a sufficient delay between stopping and starting the process so it would be good if OF also handles the no-delay use case. Thanks, Bertrand -----Original Message----- From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Luis Gomez Sent: January 27, 2017 10:52 AM To: Jamo Luhrsen <[email protected]<mailto:[email protected]>> Cc: M Vinoth <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [openflowplugin-dev] handling of OVS restart It is critical because only workaround is not acceptable :) > On Jan 27, 2017, at 10:45 AM, Jamo Luhrsen > <[email protected]<mailto:[email protected]>> wrote: > > > > On 01/26/2017 11:08 PM, Anil Vishnoi wrote: >> Bertrand, >> >> Yes the reported issue is same as you are facing with the NetVirt. >> It's happening because of the OVS don't reset/disconnect the >> connection properly (Don't send FIN/RST packet ) and netty assume >> that connection is just idle and not disconnected, so it does not >> disconnect the connection. Meanwhile if device tries to connect back, >> controller ignores that connection, because it thinks that device is already >> connection. Simple workaround is to add some minor delay (sleep .5), that >> will help with this situation. > > we can add the delays in our controlled environments to be nice, but > we can't always count on that in the wild. The workaround in the BZ > that Luis gave is to restart the controller. Seems like a drastic > workaround, imo. The bug is marked critical though, so hopefully something > can be done on our side to protect this. > > Thanks, > JamO > > >> Anil >> >> On Thu, Jan 26, 2017 at 9:35 PM, Luis Gomez >> <[email protected]<mailto:[email protected]> >> <mailto:[email protected]<mailto:[email protected]>>> wrote: >> >> Hi Bertrand, >> >> The issue I see is reproducible with just 1 controller node and it is >> similar in the sense that to it requires an OF >> channel block (e.g. as a result of an OF agent reboot) followed by an OF >> channel restore. >> >> https://bugs.opendaylight.org/show_bug.cgi?id=7689 >> <https://bugs.opendaylight.org/show_bug.cgi?id=7689> >> >> BR/Luis >> >> >>> On Jan 26, 2017, at 12:39 PM, Bertrand Low >>> <[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>> wrote: >>> >>> Hi Luis and Jozef,____ >>> __ __ >>> In our ODL cluster setup, if the device issues a “systemctl restart >>> openvswitch”, or “systemctl stop openvswitch; >>> systemctl start openvswitch” (no delay between stopping and starting), >>> we’ve observed the issue of the br-int >>> controller being unable to reconnect to the OF master node. The OF >>> master node’s role will be “other” on the device and >>> it will keep toggling between “CONNECTING” and “BACKOFF” states. The >>> other OF nodes’ roles remains as “slave”. This >>> inability to reconnect to the OF master node (and the other nodes >>> remaining as slaves and not assuming mastership) >>> appears to persist until ODL is restarted.____ >>> Luis, is this the “switch hanging” symptom you’ve mentioned?____ >>> __ __ >>> In the scenario I’ve described, we have some logs up on bug 7462 >>> (https://bugs.opendaylight.org/show_bug.cgi?id=7462 >>> <https://bugs.opendaylight.org/show_bug.cgi?id=7462>) with DEBUG enabled >>> for org.opendaylight.openflowplugin. This was >>> observed while testing legacy netvirt; we will work on providing updated >>> logs for new netvirt as well.____ >>> __ __ >>> Thanks,____ >>> Bertrand____ >>> __ __ >>> *From:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> >>> [mailto:[email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>>] >>> *On Behalf Of *Luis Gomez >>> *Sent:* January 26, 2017 9:55 AM >>> *To:* Jozef Bacigál <[email protected] >>> <mailto:[email protected]<mailto:[email protected]>>> >>> *Cc:* M Vinoth <[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>>; >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Subject:* Re: [openflowplugin-dev] handling of OVS restart____ >>> __ __ >>> Jozef and all,____ >>> __ __ >>> FYI I have seen sporadic issues of switch hanging in controller after >>> switch connection is not properly closed, I am >>> currently working to see if I can consistently reproduce these issue in >>> the ODL CI.____ >>> __ __ >>> BR/Luis____ >>> __ __ >>> __ __ >>> >>> On Jan 26, 2017, at 12:23 AM, Jozef Bacigál >>> <[email protected] >>> <mailto:[email protected]<mailto:[email protected]>>> >>> wrote:____ >>> __ __ >>> May I ask you to run this non-working case again but with ____ >>> __ __ >>> org.opendaylight.openflowplugin.impl DEBUG____ >>> __ __ >>> and send me the logs please ?____ >>> __ __ >>> Thanks____ >>> __ __ >>> Jozef____ >>> __ __ >>> __ __ >>> >>> ----------------------------------------------------------------------------------------------------------------------------- >>> *Od:* M Vinoth <[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>> >>> *Odoslané:* 25. januára 2017 15:31 >>> *Komu:* Jozef Bacigál >>> *Kópia:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Predmet:* RE: handling of OVS restart____ >>> ____ >>> Hi Jozef,____ >>> ____ >>> As you suggested I have checkout the last patch of 6802. And now I >>> can see all the changes.____ >>> ____ >>> And I have tested this changes with new netvirt build.____ >>> ____ >>> But still the dump flows are not installed properly. And other test >>> cases are working fine (1. Node disconnect 2. >>> Br-int deletion)____ >>> ____ >>> Please find my analysis in below for main difference between working >>> and failing cases,____ >>> ____ >>> *Working cases (attached in mail : try_1 document):*____ >>> ____ >>> 2017-01-25 21:14:49,043 | INFO | lt-dispatcher-17 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> Entity{type='ovsdb', >>> >>> id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b}]} >>> has no owner, cleaning up the operational data store____ >>> ____ >>> 2017-01-25 21:14:49,574 | INFO | DBConnNotifSer-5 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> InstanceIdentifier >>> KeyedInstanceIdentifier{targetType=interface >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, >>> >>> path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey >>> [_topologyId=Uri [_value=ovsdb:1]]], >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey >>> [_nodeId=Uri >>> [_value=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b]]]]} generated >>> for device connection >>> ConnectionInfo [Remote-address=10.106.138.164, Remote-port=32850, >>> Local-address10.106.138.251, Local-port=6640, >>> type=PASSIVE]____ >>> 2017-01-25 21:14:49,574 | INFO | DBConnNotifSer-5 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | OVSDB >>> entity Entity{type='ovsdb', >>> >>> id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b}]} >>> is registered for ownership.____ >>> 2017-01-25 21:14:49,692 | INFO | lt-dispatcher-17 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> handleOwnershipChanged: *this* southbound plugin instance >>> is an OWNER of the device ConnectionInfo >>> [Remote-address=10.106.138.164, Remote-port=32850, >>> Local-address10.106.138.251, Local-port=6640, type=PASSIVE]____ >>> ____ >>> *Failing Cases (attached in mail : try_2 document)*____ >>> ____ >>> 2017-01-25 21:15:38,751 | INFO | entLoopGroup-5-4 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | Library >>> disconnected PASSIVE from /10.106.138.164:32850<http://10.106.138.164:32850> >>> <http://10.106.138.164:32850> to >>> /10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640>. Cleaning up the operational >>> data store____ >>> ____ >>> 2017-01-25 21:15:39,283 | INFO | DBConnNotifSer-4 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> InstanceIdentifier >>> KeyedInstanceIdentifier{targetType=interface >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, >>> >>> path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey >>> [_topologyId=Uri [_value=ovsdb:1]]], >>> >>> org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey >>> [_nodeId=Uri >>> [_value=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b]]]]} generated >>> for device connection >>> ConnectionInfo [Remote-address=10.106.138.164, Remote-port=32862, >>> Local-address10.106.138.251, Local-port=6640, >>> type=PASSIVE]____ >>> 2017-01-25 21:15:39,284 | INFO | DBConnNotifSer-4 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | OVSDB >>> entity Entity{type='ovsdb', >>> >>> id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}]/node/node[{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/655c51fc-c86b-4cd7-bb0c-7c5d273b457b}]} >>> is registered for ownership.____ >>> 2017-01-25 21:15:39,309 | INFO | lt-dispatcher-17 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> handleOwnershipChanged: no change in ownership for >>> ConnectionInfo [Remote-address=10.106.138.164, Remote-port=32862, >>> Local-address10.106.138.251, Local-port=6640, >>> type=PASSIVE]. Ownership status is : NON-OWNER____ >>> 2017-01-25 21:15:39,333 | INFO | lt-dispatcher-19 | >>> OvsdbConnectionManager | 308 - >>> org.opendaylight.ovsdb.southbound-impl - 1.4.0.SNAPSHOT | >>> handleOwnershipChanged: *this* southbound plugin instance >>> is an OWNER of the device ConnectionInfo >>> [Remote-address=10.106.138.164, Remote-port=32862, >>> Local-address10.106.138.251, Local-port=6640, type=PASSIVE]____ >>> ____ >>> *Example:*____ >>> [stack@control devstack]$ sudo systemctl stop openvswitch____ >>> [stack@control devstack]$ sudo systemctl start openvswitch____ >>> [stack@control devstack]$____ >>> [stack@control devstack]$____ >>> [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows >>> br-int____ >>> OFPST_FLOW reply (OF1.3) (xid=0x2):____ >>> ____ >>> I will test more scenario’s and then I will let you know further.____ >>> ____ >>> Regards,____ >>> Vinoth____ >>> ____ >>> ____ >>> *From:* Jozef Bacigál >>> [mailto:[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>] >>> *Sent:* 25 January 2017 15:53 >>> *To:* M Vinoth <[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>> >>> *Cc:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Subject:* Re: handling of OVS restart____ >>> ____ >>> Hi Vinoth,____ >>> ____ >>> yes the patch for bug 6082 should prevent this kind of issues. We >>> trying to make some test with stop/start >>> connection without delay on this patch. I'll let you know if we were >>> able to reproduce the issue. ____ >>> ____ >>> Anyway as I said this patch shall hold all information about >>> switches for few seconds/milliseconds (configurable) >>> and tha just replace broken connection with correct one without any >>> changes in behavior.____ >>> ____ >>> I will let you know.____ >>> ____ >>> Jozef____ >>> ____ >>> ____ >>> >>> ----------------------------------------------------------------------------------------------------------------------------- >>> *Od:* M Vinoth <[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>> >>> *Odoslané:* 24. januára 2017 13:40 >>> *Komu:* Jozef Bacigál >>> *Kópia:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Predmet:* FW: handling of OVS restart____ >>> ____ >>> Hi Jozef,____ >>> ____ >>> I have seen your patch for bug:6082., and observe that implemented >>> related to the below issues.____ >>> https://git.opendaylight.org/gerrit/#/q/topic:%22Bug+6802%22 >>> <https://git.opendaylight.org/gerrit/#/q/topic:%22Bug+6802%22>____ >>> ____ >>> The issue is when OVS stop and start immediately without delay, that >>> time br-int flows were not available. And node >>> also disconnect.____ >>> And I have notice that life cycle service and Device context service >>> is not properly invoked when failed cases.____ >>> In below mail I have highlighted comparatively with working as well >>> as failed cases.____ >>> Whether the below issue is addressed in your patch.____ >>> Please validate the below mail and share your inputs on this.____ >>> Thanks.____ >>> ____ >>> Regards,____ >>> Vinoth____ >>> *From:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> >>> [mailto:[email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>>] >>> *On Behalf Of *M Vinoth >>> *Sent:* 24 January 2017 15:33 >>> *To:* Bertrand Low >>> <[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>>; Vishal Thapar >>> <[email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>>>; >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Cc:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>>; >>> >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Subject:* Re: [netvirt-dev] handling of OVS restart____ >>> ____ >>> Hi Vishal,____ >>> ____ >>> I have tested the following behaviour in new netvirt in 3-node >>> cluster,____ >>> ____ >>> The following commands requested without delay,____ >>> [stack@control devstack]$ sudo systemctl stop openvswitch____ >>> [stack@control devstack]$ sudo systemctl start openvswitch____ >>> ____ >>> And found that one of the node is disconnected from 3-node >>> cluster.____ >>> ____ >>> [stack@control devstack]$ sudo ovs-vsctl show____ >>> 655c51fc-c86b-4cd7-bb0c-7c5d273b457b____ >>> Manager "tcp:10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640>"____ >>> is_connected: true____ >>> Manager "tcp:10.106.138.248:6640<http://10.106.138.248:6640> >>> <http://10.106.138.248:6640>"____ >>> is_connected: true____ >>> Manager "tcp:10.106.138.244:6640<http://10.106.138.244:6640> >>> <http://10.106.138.244:6640>"____ >>> is_connected: true____ >>> Bridge br-int____ >>> Controller >>> "tcp:10.106.138.251:6653<http://10.106.138.251:6653> >>> <http://10.106.138.251:6653>"____ >>> Controller >>> "tcp:10.106.138.248:6653<http://10.106.138.248:6653> >>> <http://10.106.138.248:6653>"____ >>> is_connected: true____ >>> Controller >>> "tcp:10.106.138.244:6653<http://10.106.138.244:6653> >>> <http://10.106.138.244:6653>"____ >>> is_connected: true____ >>> fail_mode: secure____ >>> Port br-int____ >>> Interface br-int____ >>> type: internal____ >>> ovs_version: "2.5.0"____ >>> ____ >>> Verified that the internal bridge default flows are not >>> available.____ >>> ____ >>> [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows >>> br-int____ >>> OFPST_FLOW reply (OF1.3) (xid=0x2):____ >>> [stack@control devstack]$____ >>> ____ >>> The same behaviour in legacy netvirt too, but additionally one more >>> cases were failed in legacy that br-int is not >>> available.____ >>> ____ >>> I will share the following execution flows based on the log which >>> is analysed in legacy netvirt.____ >>> *_Working cases execution flows,_*____ >>> *ß**------------------ ------ Stopping the service >>> ------------------------------------------**a** **(A)*____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59630<http://10.106.138.164:59630> >>> <http://10.106.138.164:59630>, NodeId:openflow:35760623087526____ >>> OvsdbConnectionService - Connection closed ConnectionInfo____ >>> OvsdbConnectionManager - Library disconnected PASSIVE from >>> /10.106.138.164:57042<http://10.106.138.164:57042> >>> <http://10.106.138.164:57042> to >>> /10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640>. Cleaning up the operational data store____ >>> OvsdbConnectionManager - handleOwnershipChanged: *this* instance is >>> elected as an owner of the device____ >>> OvsdbConnectionManager - has no owner, cleaning up the operational >>> data store____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**a** **(B) *____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - processOvsdb Node Delete :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> *ß**------------------ ------ Setting Up the cluster service >>> ------------------------------------------**a** **(C)*____ >>> ConnectionAdapterImpl - Hello received / branch____ >>> DeviceManagerImpl - ConnectionEvent: Device connected to >>> controller,____ >>> SalRoleServiceImpl - SetRole called with input:SetRoleInput____ >>> SalRoleServiceImpl - Requesting state change to BECOMESLAVE____ >>> SalRoleServiceImpl - RoleChangeTask called on >>> device:openflow:35760623087526 OFPRole:BECOMESLAVE____ >>> RoleService - getGenerationIdFromDevice called for device: >>> openflow:35760623087526____ >>> LifecycleServiceImpl - Registered clustering MASTER services for >>> node openflow:35760623087526____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> LoggingHandler - RECEIVED: [id: 0x2cd517d7, >>> L:/10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640> - >>> R:/10.106.138.164:57066<http://10.106.138.164:57066> >>> <http://10.106.138.164:57066>]____ >>> RoleService - submitRoleChange onSuccess for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> OvsdbConnectionManager - Library connected PASSIVE from >>> /10.106.138.164:57066<http://10.106.138.164:57066> >>> <http://10.106.138.164:57066> to >>> /10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640>____ >>> OvsdbConnectionManager - OVSDB Connection from >>> /10.106.138.164:57066<http://10.106.138.164:57066> >>> <http://10.106.138.164:57066>____ >>> OvsdbConnectionManager - generated for device connection >>> ConnectionInfo____ >>> OvsdbConnectionManager - OVSDB entity Entity, is registered for >>> ownership.____ >>> LifecycleServiceImpl - Starting clustering MASTER services for node >>> openflow:35760623087526____ >>> DeviceContextImpl - Starting device context cluster services for >>> node openflow:35760623087526____ >>> OvsdbConnectionManager - handleOwnershipChanged: *this* southbound >>> plugin instance is an OWNER of the device >>> ConnectionInfo____ >>> OvsdbConnectionInstance - Monitoring database: Open_vSwitch____ >>> DeviceInitializationUtils - IP address of switch is: >>> /10.106.138.164:59640<http://10.106.138.164:59640> >>> <http://10.106.138.164:59640>____ >>> DeviceInitializationUtils - Static node Uri >>> [_value=openflow:35760623087526] info: OFPMPMETERFEATURES collected____ >>> StringValueObjectFactory - Instantiated factory for class >>> >>> org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.yang.types.rev130715.MacAddress____ >>> DeviceInitializationUtils - Static node Uri >>> [_value=openflow:35760623087526] info: OFPMPPORTDESC collected____ >>> SalRoleServiceImpl - SetRole called with input:____ >>> SalRoleServiceImpl - Requesting state change to BECOMEMASTER____ >>> RoleService - getGenerationIdFromDevice called for device: >>> openflow:35760623087526____ >>> StatisticsContextImpl - Starting statistics context cluster services >>> for node openflow:35760623087526____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMEMASTER____ >>> RoleService - submitRoleChange onSuccess for device:Uri >>> [_value=openflow:35760623087526], role:BECOMEMASTER____ >>> StatisticsManagerImpl - Scheduling statistics poll for device: Uri >>> [_value=openflow:35760623087526]____ >>> *<-------- Updating the internal bridge >>> ----------------------------**a** **(D)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Create :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> OF13Provider - initializeOFFlowRules: bridgeName: br-int____ >>> ControllerUpdateCommand - Register ODL controllers : >>> ControllerEntry____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> ____ >>> *_Failure cases execution flows (Without delay),_*____ >>> * *____ >>> *ß**------------------ ------ Stopping the service >>> ------------------------------------------**a** **(A)*____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59640<http://10.106.138.164:59640> >>> <http://10.106.138.164:59640>, NodeId:openflow:35760623087526____ >>> OvsdbConnectionService - Connection closed ConnectionInfo____ >>> OvsdbConnectionManager - Library disconnected PASSIVE from >>> /10.106.138.164:57066<http://10.106.138.164:57066> >>> <http://10.106.138.164:57066> to >>> /10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640>. Cleaning up the operational data store____ >>> OvsdbConnectionManager - has no owner, cleaning up the operational >>> data store____ >>> *ß**------------------ ------ Setting Up the cluster service >>> ------------------------------------------**a** **(C)*____ >>> ConnectionAdapterImpl - Hello received / branch____ >>> DeviceManagerImpl - ConnectionEvent: Device connected to controller, >>> Device:/10.106.138.164:59658<http://10.106.138.164:59658> >>> <http://10.106.138.164:59658>, NodeId:Uri >>> [_value=openflow:35760623087526]____ >>> SalRoleServiceImpl - SetRole called with input:SetRoleInput____ >>> SalRoleServiceImpl - Requesting state change to BECOMESLAVE____ >>> SalRoleServiceImpl - RoleChangeTask called on >>> device:openflow:35760623087526 OFPRole:BECOMESLAVE____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> RoleService - submitRoleChange onSuccess for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> LoggingHandler - RECEIVED: [id: 0x2610f6bd, >>> L:/10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640> - >>> R:/10.106.138.164:57084<http://10.106.138.164:57084> >>> <http://10.106.138.164:57084>]____ >>> OvsdbConnectionManager - Library connected PASSIVE from >>> /10.106.138.164:57084<http://10.106.138.164:57084> >>> <http://10.106.138.164:57084> to >>> /10.106.138.251:6640<http://10.106.138.251:6640> >>> <http://10.106.138.251:6640>____ >>> OvsdbConnectionManager - OVSDB Connection from >>> /10.106.138.164:57084<http://10.106.138.164:57084> >>> <http://10.106.138.164:57084>____ >>> OvsdbConnectionManager - generated for device connection >>> ConnectionInfo____ >>> OvsdbConnectionManager - OVSDB entity, is registered for >>> ownership.____ >>> OvsdbConnectionManager - handleOwnershipChanged: *this* southbound >>> plugin instance is an OWNER of the device >>> ConnectionInfo____ >>> OvsdbConnectionInstance - Monitoring database: Open_vSwitch____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**a** **(B)*____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - processOvsdb Node Delete :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> ControllerUpdateCommand - Register ODL controllers : {} bridges >>> detail : {}____ >>> ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for >>> OVSDB Bridge : null____ >>> BridgeRemovedCommand - Bridge Deleted:____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59658<http://10.106.138.164:59658> >>> <http://10.106.138.164:59658>, NodeId:openflow:35760623087526____ >>> OvsdbPortRemoveCommand - Bridge not found for port____ >>> *<-------- Updating the internal bridge >>> ----------------------------**a** **(D)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Create :____ >>> SouthboundImpl - addBridge: node:____ >>> SouthboundImpl - addBridge: result: true____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> OF13Provider - initializeOFFlowRules: bridgeName: br-int____ >>> BridgeUpdateCommand - Added ovsdb Bridge name: OvsdbBridgeName >>> [_value=br-int] uuid: null____ >>> ControllerUpdateCommand - Register ODL controllers : >>> ControllerEntry____ >>> ProtocolUpdateCommand - Updated ProtocolEntry : OpenFlow13 for OVSDB >>> Bridge : br-int____ >>> *ß**------------------ ------ Setting Up the cluster service >>> ------------------------------------------**a** **(C)*____ >>> ConnectionAdapterImpl - Hello received / branch____ >>> DeviceManagerImpl - ConnectionEvent: Device connected to controller, >>> Device:/10.106.138.164:59670<http://10.106.138.164:59670> >>> <http://10.106.138.164:59670>, NodeId:Uri >>> [_value=openflow:35760623087526]____ >>> SalRoleServiceImpl - Requesting state change to BECOMESLAVE____ >>> SalRoleServiceImpl - RoleChangeTask called on >>> device:openflow:35760623087526 OFPRole:BECOMESLAVE____ >>> RoleService - getGenerationIdFromDevice called for device: >>> openflow:35760623087526____ >>> LifecycleServiceImpl - Registered clustering MASTER services for >>> node openflow:35760623087526____ >>> RoleService - submitRoleChange called for device:Uri >>> [_value=openflow:35760623087526], role:BECOMESLAVE____ >>> RoleService - submitRoleChange onSuccess for device:____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**a** **(B)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> *<-------- Updating the internal bridge >>> ----------------------------**a** **(D)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> ADD for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - notifyNode : action: ADD, Node :____ >>> OF13Provider - initializeOFFlowRules: bridgeName: br-int____ >>> ControllerUpdateCommand - Register ODL controllers : {} bridges >>> detail : {}____ >>> ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for >>> OVSDB Bridge : null____ >>> BridgeRemovedCommand - Bridge Deleted:____ >>> OvsdbPortRemoveCommand - Bridge not found for port Port :____ >>> SystemNotificationsListenerImpl - ConnectionEvent: Connection closed >>> by device, Device:/10.106.138.164:59670<http://10.106.138.164:59670> >>> <http://10.106.138.164:59670>, NodeId:openflow:35760623087526____ >>> *ß**------------------ ------ Deleting the node and bridge >>> ------------------------------------------**a** **(B)*____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : PORT with action : >>> DELETE for the OVS node :____ >>> SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : >>> DELETE for the OVS node :____ >>> SouthboundImpl - deleteBridge node:____ >>> SouthboundHandler - notifyNode : action: DELETE, Node :____ >>> SouthboundHandler - Received ovsdbUpdate for : NODE with action : >>> UPDATE for the OVS node :____ >>> SouthboundHandler - processOvsdb Node Update :____ >>> NodeCacheManagerImpl - oldDpid == 0, newDpid == 0____ >>> SouthboundHandler - notifyNode : action: UPDATE, Node :____ >>> ____ >>> I have observe that the above working cases are properly executing >>> the life cycle service and device context for >>> cluster service, it means that properly maintain the context >>> information.____ >>> ____ >>> Please share your inputs.____ >>> ____ >>> Regards,____ >>> Vinoth____ >>> ____ >>> ____ >>> *From:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> >>> [mailto:[email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>>] >>> *On Behalf Of *Bertrand Low >>> *Sent:* 23 January 2017 11:02 >>> *To:* Vishal Thapar >>> <[email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>>>; >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Subject:* Re: [netvirt-dev] handling of OVS restart____ >>> ____ >>> Thanks Vishal,____ >>> ____ >>> The bridge deletion upon disconnect does seem like a potential issue >>> in legacy netvirt. I’ll do some testing on >>> this.____ >>> ____ >>> Cheers,____ >>> Bertrand____ >>> ____ >>> *From:* Vishal Thapar >>> [mailto:[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>] >>> *Sent:* January 22, 2017 7:59 PM >>> *To:* Bertrand Low >>> <[email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>>>; >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Subject:* RE: handling of OVS restart____ >>> ____ >>> Hi Bertrand,____ >>> ____ >>> Can you try with new netvirt? I am not that familiar with legacy >>> netvirt but new netvirt doesn’t have code to >>> delete bridge on disconnect. >>> https://bugs.opendaylight.org/show_bug.cgi?id=7488 >>> <https://bugs.opendaylight.org/show_bug.cgi?id=7488> fixed an issue >>> with dpnid changing which was impacting cluster >>> setups.____ >>> ____ >>> So, for now I’d say that this issue is not seen in new netvirt >>> unless proven otherwise J____ >>> ____ >>> Regards,____ >>> Vishal.____ >>> ____ >>> *From:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> >>> [mailto:[email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>>] >>> *On Behalf Of *Bertrand Low >>> *Sent:* 23 January 2017 08:36 >>> *To:* >>> [email protected]<mailto:[email protected]> >>> >>> <mailto:[email protected]<mailto:[email protected]>> >>> *Subject:* [netvirt-dev] handling of OVS restart____ >>> ____ >>> Hi all,____ >>> ____ >>> Has anyone run into issues with ODL handling OVS restart?____ >>> ____ >>> The scenario is like this:____ >>> ____ >>> 1) Connect device to an ODL cluster (i.e. at least 3), see that >>> br-int and the pipeline flows are installed >>> successfully____ >>> 2) Execute “restart openvswitch” on the device, or execute >>> “stop openvswitch; start openvswitch” (i.e. no >>> delay between starting and stopping openvswitch)____ >>> 3) Once the device reconnects to ODL, there are two failure >>> scenarios observed:____ >>> a. There is no br-int (Bug 7461)____ >>> 1. A connection flap is observed for br-int, sometimes with >>> br-int assigned a new dpnid____ >>> OR____ >>> b. br-int is installed, but without any pipeline flows (Bug >>> 7462)____ >>> ____ >>> Seems to me like the “restart openvswitch” use case should be >>> handled by ODL – please let me know if my >>> understanding is incorrect.____ >>> ____ >>> I’ve been testing with legacy netvirt but this timing issue may be >>> relevant to new netvirt as well.____ >>> If anyone has any insight on new netvirt’s handling of OVS restart, >>> please share. Thanks!____ >>> ____ >>> Bertrand____ >>> >>> >>> ::DISCLAIMER:: >>> >>> ----------------------------------------------------------------------------------------------------------------------------------------------------____ >>> The contents of this e-mail and any attachment(s) are confidential >>> and intended for the named recipient(s) only. >>> E-mail transmission is not guaranteed to be secure or error-free as >>> information could be intercepted, corrupted, >>> lost, destroyed, arrive late or incomplete, or may contain viruses >>> in transmission. The e mail and its contents >>> (with or without referred errors) shall therefore not attach any >>> liability on the originator or HCL or its affiliates. >>> Views or opinions, if any, presented in this email are solely those >>> of the author and may not necessarily reflect the >>> views or opinions of HCL or its affiliates. Any form of >>> reproduction, dissemination, copying, disclosure, >>> modification, >>> distribution and / or publication of this message without the prior >>> written consent of authorized representative of >>> HCL is strictly prohibited. If you have received this email in error >>> please delete it and notify the sender >>> immediately. >>> Before opening any email and/or attachments, please check them for >>> viruses and other defects.____ >>> >>> ----------------------------------------------------------------------------------------------------------------------------------------------------____ >>> ____ >>> Jozef*Bacigál*____ >>> Software Engineer____ >>> >>> Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia >>> R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia >>> +421 908 766 972 <tel:+421%20908%20766%20972> / >>> [email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>> >>> reception: +421 2 206 65 114 / >>> www.pantheon.tech<http://www.pantheon.tech> <http://www.pantheon.tech/>____ >>> logo____ >>> ____ >>> ____ >>> Jozef*Bacigál*____ >>> Software Engineer____ >>> >>> Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia >>> R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia >>> +421 908 766 972 <tel:+421%20908%20766%20972> / >>> [email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>> >>> reception: +421 2 206 65 114 / >>> www.pantheon.tech<http://www.pantheon.tech> <http://www.pantheon.tech/>____ >>> logo____ >>> ____ >>> _______________________________________________ >>> openflowplugin-dev mailing list >>> >>> [email protected]<mailto:[email protected]> >>> <mailto:[email protected]<mailto:[email protected]>> >>> >>> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >>> >>> <https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev> >> >> >> _______________________________________________ >> openflowplugin-dev mailing list >> >> [email protected]<mailto:[email protected]> >> >> <mailto:[email protected]<mailto:[email protected]>> >> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >> >> <https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev> >> >> >> >> >> -- >> Thanks >> Anil >> >> >> _______________________________________________ >> openflowplugin-dev mailing list >> [email protected]<mailto:[email protected]> >> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev >> _______________________________________________ openflowplugin-dev mailing list [email protected]<mailto:[email protected]> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev _______________________________________________ openflowplugin-dev mailing list [email protected]<mailto:[email protected]> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev JozefBacigál Software Engineer Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia +421 908 766 972 / [email protected] reception: +421 2 206 65 114 / www.pantheon.tech [logo] _______________________________________________ openflowplugin-dev mailing list [email protected]<mailto:[email protected]> https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev JozefBacigál Software Engineer Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia +421 908 766 972 / [email protected] reception: +421 2 206 65 114 / www.pantheon.tech [logo]
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
