Hi Vishal,

I have tested the following behaviour in new netvirt in 3-node cluster,

The following commands requested without delay,
[stack@control devstack]$ sudo systemctl stop openvswitch
[stack@control devstack]$ sudo systemctl start openvswitch

And found that one of the node is disconnected from 3-node cluster.

[stack@control devstack]$ sudo ovs-vsctl show
655c51fc-c86b-4cd7-bb0c-7c5d273b457b
    Manager "tcp:10.106.138.251:6640"
        is_connected: true
    Manager "tcp:10.106.138.248:6640"
        is_connected: true
    Manager "tcp:10.106.138.244:6640"
        is_connected: true
    Bridge br-int
        Controller "tcp:10.106.138.251:6653"
        Controller "tcp:10.106.138.248:6653"
            is_connected: true
        Controller "tcp:10.106.138.244:6653"
            is_connected: true
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.5.0"

Verified that the internal bridge default flows are not available.

[stack@control devstack]$  sudo ovs-ofctl -O OpenFlow13 dump-flows br-int
OFPST_FLOW reply (OF1.3) (xid=0x2):
[stack@control devstack]$

The same behaviour in legacy netvirt too, but additionally one more cases were 
failed in legacy that br-int is not available.

I will share the following execution flows based on the log  which is analysed 
in legacy netvirt.
Working cases execution flows,
<-------------------- ------ Stopping the service 
--------------------------------------------> (A)
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, 
Device:/10.106.138.164:59630, NodeId:openflow:35760623087526
OvsdbConnectionService - Connection closed ConnectionInfo
 OvsdbConnectionManager - Library disconnected PASSIVE from 
/10.106.138.164:57042 to /10.106.138.251:6640. Cleaning up the operational data 
store
OvsdbConnectionManager - handleOwnershipChanged: *this* instance is elected as 
an owner of the device
OvsdbConnectionManager - has no owner, cleaning up the operational data store
<-------------------- ------ Deleting the node and bridge 
--------------------------------------------> (B)
 SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for 
the OVS node :
 SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for 
the OVS node :
 SouthboundHandler - Received ovsdbUpdate for : NODE with action : DELETE for 
the OVS node :
 SouthboundImpl - deleteBridge node:
SouthboundHandler - processOvsdb Node Delete :
SouthboundHandler - notifyNode : action: DELETE, Node  :
SouthboundHandler - notifyNode : action: DELETE, Node  :
<-------------------- ------ Setting Up the cluster service 
--------------------------------------------> (C)
ConnectionAdapterImpl - Hello received / branch
DeviceManagerImpl - ConnectionEvent: Device connected to controller,
SalRoleServiceImpl - SetRole called with input:SetRoleInput
SalRoleServiceImpl - Requesting state change to BECOMESLAVE
SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526 
OFPRole:BECOMESLAVE
RoleService - getGenerationIdFromDevice called for device: 
openflow:35760623087526
LifecycleServiceImpl - Registered clustering MASTER services for node 
openflow:35760623087526
RoleService - submitRoleChange called for device:Uri 
[_value=openflow:35760623087526], role:BECOMESLAVE
LoggingHandler - RECEIVED: [id: 0x2cd517d7, L:/10.106.138.251:6640 - 
R:/10.106.138.164:57066]
RoleService - submitRoleChange onSuccess for device:Uri 
[_value=openflow:35760623087526], role:BECOMESLAVE
OvsdbConnectionManager - Library connected PASSIVE from /10.106.138.164:57066 
to /10.106.138.251:6640
OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57066
OvsdbConnectionManager - generated for device connection ConnectionInfo
OvsdbConnectionManager - OVSDB entity Entity, is registered for ownership.
LifecycleServiceImpl - Starting clustering MASTER services for node 
openflow:35760623087526
DeviceContextImpl - Starting device context cluster services for node 
openflow:35760623087526
OvsdbConnectionManager - handleOwnershipChanged: *this* southbound plugin 
instance is an OWNER of the device ConnectionInfo
OvsdbConnectionInstance - Monitoring database: Open_vSwitch
DeviceInitializationUtils - IP address of switch is: /10.106.138.164:59640
DeviceInitializationUtils - Static node Uri [_value=openflow:35760623087526] 
info: OFPMPMETERFEATURES collected
StringValueObjectFactory - Instantiated factory for class 
org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.yang.types.rev130715.MacAddress
DeviceInitializationUtils - Static node Uri [_value=openflow:35760623087526] 
info: OFPMPPORTDESC collected
SalRoleServiceImpl -  SetRole called with input:
SalRoleServiceImpl - Requesting state change to BECOMEMASTER
RoleService - getGenerationIdFromDevice called for device: 
openflow:35760623087526
StatisticsContextImpl - Starting statistics context cluster services for node 
openflow:35760623087526
RoleService - submitRoleChange called for device:Uri 
[_value=openflow:35760623087526], role:BECOMEMASTER
RoleService - submitRoleChange onSuccess for device:Uri 
[_value=openflow:35760623087526], role:BECOMEMASTER
StatisticsManagerImpl - Scheduling statistics poll for device: Uri 
[_value=openflow:35760623087526]
<-------- Updating the internal bridge ------------------------------> (D)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : ADD for the 
OVS node :
 SouthboundHandler - processOvsdb Node Create :
 SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for 
the OVS node :
 SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the 
OVS node :
SouthboundHandler - notifyNode : action: ADD, Node  :
SouthboundHandler - notifyNode : action: ADD, Node  :
OF13Provider - initializeOFFlowRules: bridgeName: br-int
ControllerUpdateCommand - Register ODL controllers : ControllerEntry
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for 
the OVS node :
 SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node  :

Failure cases execution flows (Without delay),

<-------------------- ------ Stopping the service 
--------------------------------------------> (A)
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, 
Device:/10.106.138.164:59640, NodeId:openflow:35760623087526
OvsdbConnectionService - Connection closed ConnectionInfo
OvsdbConnectionManager - Library disconnected PASSIVE from 
/10.106.138.164:57066 to /10.106.138.251:6640. Cleaning up the operational data 
store
OvsdbConnectionManager - has no owner, cleaning up the operational data store
<-------------------- ------ Setting Up the cluster service 
--------------------------------------------> (C)
ConnectionAdapterImpl - Hello received / branch
DeviceManagerImpl - ConnectionEvent: Device connected to controller, 
Device:/10.106.138.164:59658, NodeId:Uri [_value=openflow:35760623087526]
SalRoleServiceImpl - SetRole called with input:SetRoleInput
SalRoleServiceImpl - Requesting state change to BECOMESLAVE
SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526 
OFPRole:BECOMESLAVE
RoleService - submitRoleChange called for device:Uri 
[_value=openflow:35760623087526], role:BECOMESLAVE
RoleService - submitRoleChange onSuccess for device:Uri 
[_value=openflow:35760623087526], role:BECOMESLAVE
LoggingHandler -  RECEIVED: [id: 0x2610f6bd, L:/10.106.138.251:6640 - 
R:/10.106.138.164:57084]
OvsdbConnectionManager - Library connected PASSIVE from /10.106.138.164:57084 
to /10.106.138.251:6640
OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57084
OvsdbConnectionManager - generated for device connection ConnectionInfo
OvsdbConnectionManager - OVSDB entity, is registered for ownership.
OvsdbConnectionManager - handleOwnershipChanged: *this* southbound plugin 
instance is an OWNER of the device ConnectionInfo
OvsdbConnectionInstance - Monitoring database: Open_vSwitch
<-------------------- ------ Deleting the node and bridge 
--------------------------------------------> (B)
SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for 
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for 
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : NODE with action : DELETE for 
the OVS node :
SouthboundImpl - deleteBridge node:
SouthboundHandler - processOvsdb Node Delete :
SouthboundHandler - notifyNode : action: DELETE, Node  :
SouthboundHandler - notifyNode : action: DELETE, Node  :
ControllerUpdateCommand - Register ODL controllers : {}  bridges detail : {}
ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for OVSDB Bridge : 
null
BridgeRemovedCommand - Bridge Deleted:
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, 
Device:/10.106.138.164:59658, NodeId:openflow:35760623087526
OvsdbPortRemoveCommand - Bridge not found for port
<-------- Updating the internal bridge ------------------------------> (D)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : ADD for the 
OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for the 
OVS node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the 
OVS node :
SouthboundHandler - processOvsdb Node Create :
SouthboundImpl - addBridge: node:
SouthboundImpl - addBridge: result: true
SouthboundHandler - notifyNode : action: ADD, Node  :
SouthboundHandler - notifyNode : action: ADD, Node  :
OF13Provider - initializeOFFlowRules: bridgeName: br-int
BridgeUpdateCommand - Added ovsdb Bridge name: OvsdbBridgeName [_value=br-int] 
uuid: null
ControllerUpdateCommand - Register ODL controllers : ControllerEntry
ProtocolUpdateCommand - Updated ProtocolEntry : OpenFlow13 for OVSDB Bridge : 
br-int
<-------------------- ------ Setting Up the cluster service 
--------------------------------------------> (C)
ConnectionAdapterImpl - Hello received / branch
DeviceManagerImpl - ConnectionEvent: Device connected to controller, 
Device:/10.106.138.164:59670, NodeId:Uri [_value=openflow:35760623087526]
SalRoleServiceImpl - Requesting state change to BECOMESLAVE
SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526 
OFPRole:BECOMESLAVE
RoleService - getGenerationIdFromDevice called for device: 
openflow:35760623087526
LifecycleServiceImpl - Registered clustering MASTER services for node 
openflow:35760623087526
RoleService - submitRoleChange called for device:Uri 
[_value=openflow:35760623087526], role:BECOMESLAVE
RoleService - submitRoleChange onSuccess for device:
<-------------------- ------ Deleting the node and bridge 
--------------------------------------------> (B)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for 
the OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node  :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for 
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for 
the OVS node :
SouthboundImpl - deleteBridge node:
SouthboundHandler - notifyNode : action: DELETE, Node  :
<-------- Updating the internal bridge ------------------------------> (D)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for 
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for the 
OVS node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the 
OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node  :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : UPDATE for 
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : UPDATE for 
the OVS node :
SouthboundHandler - notifyNode : action: ADD, Node  :
OF13Provider - initializeOFFlowRules: bridgeName: br-int
ControllerUpdateCommand - Register ODL controllers : {}  bridges detail : {}
ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for OVSDB Bridge : 
null
BridgeRemovedCommand - Bridge Deleted:
OvsdbPortRemoveCommand - Bridge not found for port Port :
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device, 
Device:/10.106.138.164:59670, NodeId:openflow:35760623087526
<-------------------- ------ Deleting the node and bridge 
--------------------------------------------> (B)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for 
the OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node  :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for 
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for 
the OVS node :
SouthboundImpl - deleteBridge node:
SouthboundHandler - notifyNode : action: DELETE, Node  :
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for 
the OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node  :

I have observe that the above working cases are properly executing the life 
cycle service and device context for cluster service, it means that properly 
maintain the context information.

Please share your inputs.

Regards,
Vinoth


From: netvirt-dev-boun...@lists.opendaylight.org 
[mailto:netvirt-dev-boun...@lists.opendaylight.org] On Behalf Of Bertrand Low
Sent: 23 January 2017 11:02
To: Vishal Thapar <vishal.tha...@ericsson.com>; 
netvirt-...@lists.opendaylight.org
Subject: Re: [netvirt-dev] handling of OVS restart

Thanks Vishal,

The bridge deletion upon disconnect does seem like a potential issue in legacy 
netvirt. I'll do some testing on this.

Cheers,
Bertrand

From: Vishal Thapar [mailto:vishal.tha...@ericsson.com]
Sent: January 22, 2017 7:59 PM
To: Bertrand Low <bertrand....@hcl.com<mailto:bertrand....@hcl.com>>; 
netvirt-...@lists.opendaylight.org<mailto:netvirt-...@lists.opendaylight.org>
Subject: RE: handling of OVS restart

Hi Bertrand,

Can you try with new netvirt? I am not that familiar with legacy netvirt but 
new netvirt doesn't have code to delete bridge on disconnect. 
https://bugs.opendaylight.org/show_bug.cgi?id=7488 fixed an issue with dpnid 
changing which was impacting cluster setups.

So, for now I'd say that this issue is not seen in new netvirt unless proven 
otherwise :)

Regards,
Vishal.

From: 
netvirt-dev-boun...@lists.opendaylight.org<mailto:netvirt-dev-boun...@lists.opendaylight.org>
 [mailto:netvirt-dev-boun...@lists.opendaylight.org] On Behalf Of Bertrand Low
Sent: 23 January 2017 08:36
To: 
netvirt-...@lists.opendaylight.org<mailto:netvirt-...@lists.opendaylight.org>
Subject: [netvirt-dev] handling of OVS restart

Hi all,

Has anyone run into issues with ODL handling OVS restart?

The scenario is like this:


1)      Connect device to an ODL cluster (i.e. at least 3), see that br-int and 
the pipeline flows are installed successfully

2)      Execute "restart openvswitch" on the device, or execute "stop 
openvswitch; start openvswitch" (i.e. no delay between starting and stopping 
openvswitch)

3)      Once the device reconnects to ODL, there are two failure scenarios 
observed:

a.       There is no br-int (Bug 7461)

1.       A connection flap is observed for br-int, sometimes with br-int 
assigned a new dpnid
OR

b.      br-int is installed, but without any pipeline flows (Bug 7462)

Seems to me like the "restart openvswitch" use case should be handled by ODL - 
please let me know if my understanding is incorrect.

I've been testing with legacy netvirt but this timing issue may be relevant to 
new netvirt as well.
If anyone has any insight on new netvirt's handling of OVS restart, please 
share. Thanks!

Bertrand


::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended 
for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information 
could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in 
transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on 
the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the 
author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, 
dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written 
consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please 
delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and 
other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev

Reply via email to