Hi Vishal, I have tested the following behaviour in new netvirt in 3-node cluster,
The following commands requested without delay, [stack@control devstack]$ sudo systemctl stop openvswitch [stack@control devstack]$ sudo systemctl start openvswitch And found that one of the node is disconnected from 3-node cluster. [stack@control devstack]$ sudo ovs-vsctl show 655c51fc-c86b-4cd7-bb0c-7c5d273b457b Manager "tcp:10.106.138.251:6640" is_connected: true Manager "tcp:10.106.138.248:6640" is_connected: true Manager "tcp:10.106.138.244:6640" is_connected: true Bridge br-int Controller "tcp:10.106.138.251:6653" Controller "tcp:10.106.138.248:6653" is_connected: true Controller "tcp:10.106.138.244:6653" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal ovs_version: "2.5.0" Verified that the internal bridge default flows are not available. [stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int OFPST_FLOW reply (OF1.3) (xid=0x2): [stack@control devstack]$ The same behaviour in legacy netvirt too, but additionally one more cases were failed in legacy that br-int is not available. I will share the following execution flows based on the log which is analysed in legacy netvirt. Working cases execution flows, <-------------------- ------ Stopping the service --------------------------------------------> (A) SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, Device:/10.106.138.164:59630, NodeId:openflow:35760623087526 OvsdbConnectionService - Connection closed ConnectionInfo OvsdbConnectionManager - Library disconnected PASSIVE from /10.106.138.164:57042 to /10.106.138.251:6640. Cleaning up the operational data store OvsdbConnectionManager - handleOwnershipChanged: *this* instance is elected as an owner of the device OvsdbConnectionManager - has no owner, cleaning up the operational data store <-------------------- ------ Deleting the node and bridge --------------------------------------------> (B) SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : NODE with action : DELETE for the OVS node : SouthboundImpl - deleteBridge node: SouthboundHandler - processOvsdb Node Delete : SouthboundHandler - notifyNode : action: DELETE, Node : SouthboundHandler - notifyNode : action: DELETE, Node : <-------------------- ------ Setting Up the cluster service --------------------------------------------> (C) ConnectionAdapterImpl - Hello received / branch DeviceManagerImpl - ConnectionEvent: Device connected to controller, SalRoleServiceImpl - SetRole called with input:SetRoleInput SalRoleServiceImpl - Requesting state change to BECOMESLAVE SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526 OFPRole:BECOMESLAVE RoleService - getGenerationIdFromDevice called for device: openflow:35760623087526 LifecycleServiceImpl - Registered clustering MASTER services for node openflow:35760623087526 RoleService - submitRoleChange called for device:Uri [_value=openflow:35760623087526], role:BECOMESLAVE LoggingHandler - RECEIVED: [id: 0x2cd517d7, L:/10.106.138.251:6640 - R:/10.106.138.164:57066] RoleService - submitRoleChange onSuccess for device:Uri [_value=openflow:35760623087526], role:BECOMESLAVE OvsdbConnectionManager - Library connected PASSIVE from /10.106.138.164:57066 to /10.106.138.251:6640 OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57066 OvsdbConnectionManager - generated for device connection ConnectionInfo OvsdbConnectionManager - OVSDB entity Entity, is registered for ownership. LifecycleServiceImpl - Starting clustering MASTER services for node openflow:35760623087526 DeviceContextImpl - Starting device context cluster services for node openflow:35760623087526 OvsdbConnectionManager - handleOwnershipChanged: *this* southbound plugin instance is an OWNER of the device ConnectionInfo OvsdbConnectionInstance - Monitoring database: Open_vSwitch DeviceInitializationUtils - IP address of switch is: /10.106.138.164:59640 DeviceInitializationUtils - Static node Uri [_value=openflow:35760623087526] info: OFPMPMETERFEATURES collected StringValueObjectFactory - Instantiated factory for class org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.yang.types.rev130715.MacAddress DeviceInitializationUtils - Static node Uri [_value=openflow:35760623087526] info: OFPMPPORTDESC collected SalRoleServiceImpl - SetRole called with input: SalRoleServiceImpl - Requesting state change to BECOMEMASTER RoleService - getGenerationIdFromDevice called for device: openflow:35760623087526 StatisticsContextImpl - Starting statistics context cluster services for node openflow:35760623087526 RoleService - submitRoleChange called for device:Uri [_value=openflow:35760623087526], role:BECOMEMASTER RoleService - submitRoleChange onSuccess for device:Uri [_value=openflow:35760623087526], role:BECOMEMASTER StatisticsManagerImpl - Scheduling statistics poll for device: Uri [_value=openflow:35760623087526] <-------- Updating the internal bridge ------------------------------> (D) SouthboundHandler - Received ovsdbUpdate for : NODE with action : ADD for the OVS node : SouthboundHandler - processOvsdb Node Create : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for the OVS node : SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the OVS node : SouthboundHandler - notifyNode : action: ADD, Node : SouthboundHandler - notifyNode : action: ADD, Node : OF13Provider - initializeOFFlowRules: bridgeName: br-int ControllerUpdateCommand - Register ODL controllers : ControllerEntry SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for the OVS node : SouthboundHandler - processOvsdb Node Update : NodeCacheManagerImpl - oldDpid == 0, newDpid == 0 SouthboundHandler - notifyNode : action: UPDATE, Node : Failure cases execution flows (Without delay), <-------------------- ------ Stopping the service --------------------------------------------> (A) SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, Device:/10.106.138.164:59640, NodeId:openflow:35760623087526 OvsdbConnectionService - Connection closed ConnectionInfo OvsdbConnectionManager - Library disconnected PASSIVE from /10.106.138.164:57066 to /10.106.138.251:6640. Cleaning up the operational data store OvsdbConnectionManager - has no owner, cleaning up the operational data store <-------------------- ------ Setting Up the cluster service --------------------------------------------> (C) ConnectionAdapterImpl - Hello received / branch DeviceManagerImpl - ConnectionEvent: Device connected to controller, Device:/10.106.138.164:59658, NodeId:Uri [_value=openflow:35760623087526] SalRoleServiceImpl - SetRole called with input:SetRoleInput SalRoleServiceImpl - Requesting state change to BECOMESLAVE SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526 OFPRole:BECOMESLAVE RoleService - submitRoleChange called for device:Uri [_value=openflow:35760623087526], role:BECOMESLAVE RoleService - submitRoleChange onSuccess for device:Uri [_value=openflow:35760623087526], role:BECOMESLAVE LoggingHandler - RECEIVED: [id: 0x2610f6bd, L:/10.106.138.251:6640 - R:/10.106.138.164:57084] OvsdbConnectionManager - Library connected PASSIVE from /10.106.138.164:57084 to /10.106.138.251:6640 OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57084 OvsdbConnectionManager - generated for device connection ConnectionInfo OvsdbConnectionManager - OVSDB entity, is registered for ownership. OvsdbConnectionManager - handleOwnershipChanged: *this* southbound plugin instance is an OWNER of the device ConnectionInfo OvsdbConnectionInstance - Monitoring database: Open_vSwitch <-------------------- ------ Deleting the node and bridge --------------------------------------------> (B) SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : NODE with action : DELETE for the OVS node : SouthboundImpl - deleteBridge node: SouthboundHandler - processOvsdb Node Delete : SouthboundHandler - notifyNode : action: DELETE, Node : SouthboundHandler - notifyNode : action: DELETE, Node : ControllerUpdateCommand - Register ODL controllers : {} bridges detail : {} ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for OVSDB Bridge : null BridgeRemovedCommand - Bridge Deleted: SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, Device:/10.106.138.164:59658, NodeId:openflow:35760623087526 OvsdbPortRemoveCommand - Bridge not found for port <-------- Updating the internal bridge ------------------------------> (D) SouthboundHandler - Received ovsdbUpdate for : NODE with action : ADD for the OVS node : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for the OVS node : SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the OVS node : SouthboundHandler - processOvsdb Node Create : SouthboundImpl - addBridge: node: SouthboundImpl - addBridge: result: true SouthboundHandler - notifyNode : action: ADD, Node : SouthboundHandler - notifyNode : action: ADD, Node : OF13Provider - initializeOFFlowRules: bridgeName: br-int BridgeUpdateCommand - Added ovsdb Bridge name: OvsdbBridgeName [_value=br-int] uuid: null ControllerUpdateCommand - Register ODL controllers : ControllerEntry ProtocolUpdateCommand - Updated ProtocolEntry : OpenFlow13 for OVSDB Bridge : br-int <-------------------- ------ Setting Up the cluster service --------------------------------------------> (C) ConnectionAdapterImpl - Hello received / branch DeviceManagerImpl - ConnectionEvent: Device connected to controller, Device:/10.106.138.164:59670, NodeId:Uri [_value=openflow:35760623087526] SalRoleServiceImpl - Requesting state change to BECOMESLAVE SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526 OFPRole:BECOMESLAVE RoleService - getGenerationIdFromDevice called for device: openflow:35760623087526 LifecycleServiceImpl - Registered clustering MASTER services for node openflow:35760623087526 RoleService - submitRoleChange called for device:Uri [_value=openflow:35760623087526], role:BECOMESLAVE RoleService - submitRoleChange onSuccess for device: <-------------------- ------ Deleting the node and bridge --------------------------------------------> (B) SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for the OVS node : SouthboundHandler - processOvsdb Node Update : NodeCacheManagerImpl - oldDpid == 0, newDpid == 0 SouthboundHandler - notifyNode : action: UPDATE, Node : SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for the OVS node : SouthboundImpl - deleteBridge node: SouthboundHandler - notifyNode : action: DELETE, Node : <-------- Updating the internal bridge ------------------------------> (D) SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for the OVS node : SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the OVS node : SouthboundHandler - processOvsdb Node Update : NodeCacheManagerImpl - oldDpid == 0, newDpid == 0 SouthboundHandler - notifyNode : action: UPDATE, Node : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : UPDATE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : PORT with action : UPDATE for the OVS node : SouthboundHandler - notifyNode : action: ADD, Node : OF13Provider - initializeOFFlowRules: bridgeName: br-int ControllerUpdateCommand - Register ODL controllers : {} bridges detail : {} ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for OVSDB Bridge : null BridgeRemovedCommand - Bridge Deleted: OvsdbPortRemoveCommand - Bridge not found for port Port : SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, Device:/10.106.138.164:59670, NodeId:openflow:35760623087526 <-------------------- ------ Deleting the node and bridge --------------------------------------------> (B) SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for the OVS node : SouthboundHandler - processOvsdb Node Update : NodeCacheManagerImpl - oldDpid == 0, newDpid == 0 SouthboundHandler - notifyNode : action: UPDATE, Node : SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for the OVS node : SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for the OVS node : SouthboundImpl - deleteBridge node: SouthboundHandler - notifyNode : action: DELETE, Node : SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for the OVS node : SouthboundHandler - processOvsdb Node Update : NodeCacheManagerImpl - oldDpid == 0, newDpid == 0 SouthboundHandler - notifyNode : action: UPDATE, Node : I have observe that the above working cases are properly executing the life cycle service and device context for cluster service, it means that properly maintain the context information. Please share your inputs. Regards, Vinoth From: netvirt-dev-boun...@lists.opendaylight.org [mailto:netvirt-dev-boun...@lists.opendaylight.org] On Behalf Of Bertrand Low Sent: 23 January 2017 11:02 To: Vishal Thapar <vishal.tha...@ericsson.com>; netvirt-...@lists.opendaylight.org Subject: Re: [netvirt-dev] handling of OVS restart Thanks Vishal, The bridge deletion upon disconnect does seem like a potential issue in legacy netvirt. I'll do some testing on this. Cheers, Bertrand From: Vishal Thapar [mailto:vishal.tha...@ericsson.com] Sent: January 22, 2017 7:59 PM To: Bertrand Low <bertrand....@hcl.com<mailto:bertrand....@hcl.com>>; netvirt-...@lists.opendaylight.org<mailto:netvirt-...@lists.opendaylight.org> Subject: RE: handling of OVS restart Hi Bertrand, Can you try with new netvirt? I am not that familiar with legacy netvirt but new netvirt doesn't have code to delete bridge on disconnect. https://bugs.opendaylight.org/show_bug.cgi?id=7488 fixed an issue with dpnid changing which was impacting cluster setups. So, for now I'd say that this issue is not seen in new netvirt unless proven otherwise :) Regards, Vishal. From: netvirt-dev-boun...@lists.opendaylight.org<mailto:netvirt-dev-boun...@lists.opendaylight.org> [mailto:netvirt-dev-boun...@lists.opendaylight.org] On Behalf Of Bertrand Low Sent: 23 January 2017 08:36 To: netvirt-...@lists.opendaylight.org<mailto:netvirt-...@lists.opendaylight.org> Subject: [netvirt-dev] handling of OVS restart Hi all, Has anyone run into issues with ODL handling OVS restart? The scenario is like this: 1) Connect device to an ODL cluster (i.e. at least 3), see that br-int and the pipeline flows are installed successfully 2) Execute "restart openvswitch" on the device, or execute "stop openvswitch; start openvswitch" (i.e. no delay between starting and stopping openvswitch) 3) Once the device reconnects to ODL, there are two failure scenarios observed: a. There is no br-int (Bug 7461) 1. A connection flap is observed for br-int, sometimes with br-int assigned a new dpnid OR b. br-int is installed, but without any pipeline flows (Bug 7462) Seems to me like the "restart openvswitch" use case should be handled by ODL - please let me know if my understanding is incorrect. I've been testing with legacy netvirt but this timing issue may be relevant to new netvirt as well. If anyone has any insight on new netvirt's handling of OVS restart, please share. Thanks! Bertrand ::DISCLAIMER:: ---------------------------------------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. ----------------------------------------------------------------------------------------------------------------------------------------------------
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev