Hi Vishal,
I have tested the following behaviour in new netvirt in 3-node cluster,
The following commands requested without delay,
[stack@control devstack]$ sudo systemctl stop openvswitch
[stack@control devstack]$ sudo systemctl start openvswitch
And found that one of the node is disconnected from 3-node cluster.
[stack@control devstack]$ sudo ovs-vsctl show
655c51fc-c86b-4cd7-bb0c-7c5d273b457b
Manager "tcp:10.106.138.251:6640"
is_connected: true
Manager "tcp:10.106.138.248:6640"
is_connected: true
Manager "tcp:10.106.138.244:6640"
is_connected: true
Bridge br-int
Controller "tcp:10.106.138.251:6653"
Controller "tcp:10.106.138.248:6653"
is_connected: true
Controller "tcp:10.106.138.244:6653"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
ovs_version: "2.5.0"
Verified that the internal bridge default flows are not available.
[stack@control devstack]$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int
OFPST_FLOW reply (OF1.3) (xid=0x2):
[stack@control devstack]$
The same behaviour in legacy netvirt too, but additionally one more cases were
failed in legacy that br-int is not available.
I will share the following execution flows based on the log which is analysed
in legacy netvirt.
Working cases execution flows,
<-------------------- ------ Stopping the service
--------------------------------------------> (A)
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device,
Device:/10.106.138.164:59630, NodeId:openflow:35760623087526
OvsdbConnectionService - Connection closed ConnectionInfo
OvsdbConnectionManager - Library disconnected PASSIVE from
/10.106.138.164:57042 to /10.106.138.251:6640. Cleaning up the operational data
store
OvsdbConnectionManager - handleOwnershipChanged: *this* instance is elected as
an owner of the device
OvsdbConnectionManager - has no owner, cleaning up the operational data store
<-------------------- ------ Deleting the node and bridge
--------------------------------------------> (B)
SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : NODE with action : DELETE for
the OVS node :
SouthboundImpl - deleteBridge node:
SouthboundHandler - processOvsdb Node Delete :
SouthboundHandler - notifyNode : action: DELETE, Node :
SouthboundHandler - notifyNode : action: DELETE, Node :
<-------------------- ------ Setting Up the cluster service
--------------------------------------------> (C)
ConnectionAdapterImpl - Hello received / branch
DeviceManagerImpl - ConnectionEvent: Device connected to controller,
SalRoleServiceImpl - SetRole called with input:SetRoleInput
SalRoleServiceImpl - Requesting state change to BECOMESLAVE
SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526
OFPRole:BECOMESLAVE
RoleService - getGenerationIdFromDevice called for device:
openflow:35760623087526
LifecycleServiceImpl - Registered clustering MASTER services for node
openflow:35760623087526
RoleService - submitRoleChange called for device:Uri
[_value=openflow:35760623087526], role:BECOMESLAVE
LoggingHandler - RECEIVED: [id: 0x2cd517d7, L:/10.106.138.251:6640 -
R:/10.106.138.164:57066]
RoleService - submitRoleChange onSuccess for device:Uri
[_value=openflow:35760623087526], role:BECOMESLAVE
OvsdbConnectionManager - Library connected PASSIVE from /10.106.138.164:57066
to /10.106.138.251:6640
OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57066
OvsdbConnectionManager - generated for device connection ConnectionInfo
OvsdbConnectionManager - OVSDB entity Entity, is registered for ownership.
LifecycleServiceImpl - Starting clustering MASTER services for node
openflow:35760623087526
DeviceContextImpl - Starting device context cluster services for node
openflow:35760623087526
OvsdbConnectionManager - handleOwnershipChanged: *this* southbound plugin
instance is an OWNER of the device ConnectionInfo
OvsdbConnectionInstance - Monitoring database: Open_vSwitch
DeviceInitializationUtils - IP address of switch is: /10.106.138.164:59640
DeviceInitializationUtils - Static node Uri [_value=openflow:35760623087526]
info: OFPMPMETERFEATURES collected
StringValueObjectFactory - Instantiated factory for class
org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.yang.types.rev130715.MacAddress
DeviceInitializationUtils - Static node Uri [_value=openflow:35760623087526]
info: OFPMPPORTDESC collected
SalRoleServiceImpl - SetRole called with input:
SalRoleServiceImpl - Requesting state change to BECOMEMASTER
RoleService - getGenerationIdFromDevice called for device:
openflow:35760623087526
StatisticsContextImpl - Starting statistics context cluster services for node
openflow:35760623087526
RoleService - submitRoleChange called for device:Uri
[_value=openflow:35760623087526], role:BECOMEMASTER
RoleService - submitRoleChange onSuccess for device:Uri
[_value=openflow:35760623087526], role:BECOMEMASTER
StatisticsManagerImpl - Scheduling statistics poll for device: Uri
[_value=openflow:35760623087526]
<-------- Updating the internal bridge ------------------------------> (D)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : ADD for the
OVS node :
SouthboundHandler - processOvsdb Node Create :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the
OVS node :
SouthboundHandler - notifyNode : action: ADD, Node :
SouthboundHandler - notifyNode : action: ADD, Node :
OF13Provider - initializeOFFlowRules: bridgeName: br-int
ControllerUpdateCommand - Register ODL controllers : ControllerEntry
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for
the OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node :
Failure cases execution flows (Without delay),
<-------------------- ------ Stopping the service
--------------------------------------------> (A)
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device,
Device:/10.106.138.164:59640, NodeId:openflow:35760623087526
OvsdbConnectionService - Connection closed ConnectionInfo
OvsdbConnectionManager - Library disconnected PASSIVE from
/10.106.138.164:57066 to /10.106.138.251:6640. Cleaning up the operational data
store
OvsdbConnectionManager - has no owner, cleaning up the operational data store
<-------------------- ------ Setting Up the cluster service
--------------------------------------------> (C)
ConnectionAdapterImpl - Hello received / branch
DeviceManagerImpl - ConnectionEvent: Device connected to controller,
Device:/10.106.138.164:59658, NodeId:Uri [_value=openflow:35760623087526]
SalRoleServiceImpl - SetRole called with input:SetRoleInput
SalRoleServiceImpl - Requesting state change to BECOMESLAVE
SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526
OFPRole:BECOMESLAVE
RoleService - submitRoleChange called for device:Uri
[_value=openflow:35760623087526], role:BECOMESLAVE
RoleService - submitRoleChange onSuccess for device:Uri
[_value=openflow:35760623087526], role:BECOMESLAVE
LoggingHandler - RECEIVED: [id: 0x2610f6bd, L:/10.106.138.251:6640 -
R:/10.106.138.164:57084]
OvsdbConnectionManager - Library connected PASSIVE from /10.106.138.164:57084
to /10.106.138.251:6640
OvsdbConnectionManager - OVSDB Connection from /10.106.138.164:57084
OvsdbConnectionManager - generated for device connection ConnectionInfo
OvsdbConnectionManager - OVSDB entity, is registered for ownership.
OvsdbConnectionManager - handleOwnershipChanged: *this* southbound plugin
instance is an OWNER of the device ConnectionInfo
OvsdbConnectionInstance - Monitoring database: Open_vSwitch
<-------------------- ------ Deleting the node and bridge
--------------------------------------------> (B)
SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : NODE with action : DELETE for
the OVS node :
SouthboundImpl - deleteBridge node:
SouthboundHandler - processOvsdb Node Delete :
SouthboundHandler - notifyNode : action: DELETE, Node :
SouthboundHandler - notifyNode : action: DELETE, Node :
ControllerUpdateCommand - Register ODL controllers : {} bridges detail : {}
ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for OVSDB Bridge :
null
BridgeRemovedCommand - Bridge Deleted:
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device,
Device:/10.106.138.164:59658, NodeId:openflow:35760623087526
OvsdbPortRemoveCommand - Bridge not found for port
<-------- Updating the internal bridge ------------------------------> (D)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : ADD for the
OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for the
OVS node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the
OVS node :
SouthboundHandler - processOvsdb Node Create :
SouthboundImpl - addBridge: node:
SouthboundImpl - addBridge: result: true
SouthboundHandler - notifyNode : action: ADD, Node :
SouthboundHandler - notifyNode : action: ADD, Node :
OF13Provider - initializeOFFlowRules: bridgeName: br-int
BridgeUpdateCommand - Added ovsdb Bridge name: OvsdbBridgeName [_value=br-int]
uuid: null
ControllerUpdateCommand - Register ODL controllers : ControllerEntry
ProtocolUpdateCommand - Updated ProtocolEntry : OpenFlow13 for OVSDB Bridge :
br-int
<-------------------- ------ Setting Up the cluster service
--------------------------------------------> (C)
ConnectionAdapterImpl - Hello received / branch
DeviceManagerImpl - ConnectionEvent: Device connected to controller,
Device:/10.106.138.164:59670, NodeId:Uri [_value=openflow:35760623087526]
SalRoleServiceImpl - Requesting state change to BECOMESLAVE
SalRoleServiceImpl - RoleChangeTask called on device:openflow:35760623087526
OFPRole:BECOMESLAVE
RoleService - getGenerationIdFromDevice called for device:
openflow:35760623087526
LifecycleServiceImpl - Registered clustering MASTER services for node
openflow:35760623087526
RoleService - submitRoleChange called for device:Uri
[_value=openflow:35760623087526], role:BECOMESLAVE
RoleService - submitRoleChange onSuccess for device:
<-------------------- ------ Deleting the node and bridge
--------------------------------------------> (B)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for
the OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for
the OVS node :
SouthboundImpl - deleteBridge node:
SouthboundHandler - notifyNode : action: DELETE, Node :
<-------- Updating the internal bridge ------------------------------> (D)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : ADD for the
OVS node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : ADD for the
OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : UPDATE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : UPDATE for
the OVS node :
SouthboundHandler - notifyNode : action: ADD, Node :
OF13Provider - initializeOFFlowRules: bridgeName: br-int
ControllerUpdateCommand - Register ODL controllers : {} bridges detail : {}
ProtocolRemovedCommand - Removed ProtocolEntry : OpenFlow13 for OVSDB Bridge :
null
BridgeRemovedCommand - Bridge Deleted:
OvsdbPortRemoveCommand - Bridge not found for port Port :
SystemNotificationsListenerImpl - ConnectionEvent: Connection closed by device,
Device:/10.106.138.164:59670, NodeId:openflow:35760623087526
<-------------------- ------ Deleting the node and bridge
--------------------------------------------> (B)
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for
the OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node :
SouthboundHandler - Received ovsdbUpdate for : PORT with action : DELETE for
the OVS node :
SouthboundHandler - Received ovsdbUpdate for : BRIDGE with action : DELETE for
the OVS node :
SouthboundImpl - deleteBridge node:
SouthboundHandler - notifyNode : action: DELETE, Node :
SouthboundHandler - Received ovsdbUpdate for : NODE with action : UPDATE for
the OVS node :
SouthboundHandler - processOvsdb Node Update :
NodeCacheManagerImpl - oldDpid == 0, newDpid == 0
SouthboundHandler - notifyNode : action: UPDATE, Node :
I have observe that the above working cases are properly executing the life
cycle service and device context for cluster service, it means that properly
maintain the context information.
Please share your inputs.
Regards,
Vinoth
From: [email protected]
[mailto:[email protected]] On Behalf Of Bertrand Low
Sent: 23 January 2017 11:02
To: Vishal Thapar <[email protected]>;
[email protected]
Subject: Re: [netvirt-dev] handling of OVS restart
Thanks Vishal,
The bridge deletion upon disconnect does seem like a potential issue in legacy
netvirt. I'll do some testing on this.
Cheers,
Bertrand
From: Vishal Thapar [mailto:[email protected]]
Sent: January 22, 2017 7:59 PM
To: Bertrand Low <[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: RE: handling of OVS restart
Hi Bertrand,
Can you try with new netvirt? I am not that familiar with legacy netvirt but
new netvirt doesn't have code to delete bridge on disconnect.
https://bugs.opendaylight.org/show_bug.cgi?id=7488 fixed an issue with dpnid
changing which was impacting cluster setups.
So, for now I'd say that this issue is not seen in new netvirt unless proven
otherwise :)
Regards,
Vishal.
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Bertrand Low
Sent: 23 January 2017 08:36
To:
[email protected]<mailto:[email protected]>
Subject: [netvirt-dev] handling of OVS restart
Hi all,
Has anyone run into issues with ODL handling OVS restart?
The scenario is like this:
1) Connect device to an ODL cluster (i.e. at least 3), see that br-int and
the pipeline flows are installed successfully
2) Execute "restart openvswitch" on the device, or execute "stop
openvswitch; start openvswitch" (i.e. no delay between starting and stopping
openvswitch)
3) Once the device reconnects to ODL, there are two failure scenarios
observed:
a. There is no br-int (Bug 7461)
1. A connection flap is observed for br-int, sometimes with br-int
assigned a new dpnid
OR
b. br-int is installed, but without any pipeline flows (Bug 7462)
Seems to me like the "restart openvswitch" use case should be handled by ODL -
please let me know if my understanding is incorrect.
I've been testing with legacy netvirt but this timing issue may be relevant to
new netvirt as well.
If anyone has any insight on new netvirt's handling of OVS restart, please
share. Thanks!
Bertrand
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended
for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information
could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in
transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on
the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the
author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written
consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please
delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and
other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
_______________________________________________
openflowplugin-dev mailing list
[email protected]
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
