Last week’s discussion on reconciliation/upgrade profiles,
1. There shall be two profiles/modes configured which can change OFP behavior
in context of reconciliation
Reconciliation Mode : <ARBITARTED|AUTONOMOUS>
AUTONOMOUS mode implies that OFPlugin shall perform reconciliation
autonomously as it does now without any change in the workflow - ie. Switch-
connection-handling followed by bundle-based Reconciliaion execution
followed by publishing of switch to the Inventory Opertaional DS
ARBITRATED mode implies that OFPlugin shall not execute reconciliation by
itself but by allowing higher level app "Reconciliation arbiter / orchestrator"
to initiate and
complete the bundle-based reconciliation including any error-handling thereof
2. This implies two changes for OFPlugin
a) OFP must publish the switch to Inventory Operational soon after the
connection-handling and master-election phases complete without triggering
Reconciliation by
itself in ARBITRATED mode
b) OFP must expose RPCs to start, modify and close bundle so that
"Reconciliation arbiter / orchestrator" is able to invoke the same to realize
reconciliation
Reconciliation Arbiter / Orchestrator (which is functional only in scenario
when Reconciliation-Mode==ARBITRATED)
============================================================================
Listeners for Reconciliation Arbiter / Orchestrator :
This is more or less similar to current reconciliation function of FRM
a) Inventory Oper DTCN for switch status sensing
b) Config Inventory DTCN for flows / groups and meters
Conditions:
------------
a) Whether or not data is available for given switch in Config Inventory DS
b) Whether the system is in upgrade mode
Common reaction for Switch Connection handling when
- Condition 1 : UPGRADE_MODE=true
- Condition 2 : UPGRADE_MODE=false
For all conditions, reconciliation can be executed by the arbiter as per below
steps
Trigger reconciliation by
a) Issuing implicit close of implicit bundle-id - this is mainly to
address following failure scenario in OF-HA context : For OFHA purge the
incomplete implicit bundles
which might have been started by previous switch-master in case
leadership changes. Since we use a single implicit bundle-id, issuing a
cancel-bundle from new
switch-master implicitly would not result in any issues and can be
considered as safety measure.
b) starting implicit bundle using OFP RPC
b) reading flows / groups / meters from DS if present
c) add the same to the implicit bundle if available
d) defer the bundle close for a period of time (which is not a timer
managed by OFP but by some static configuration managed within the scope of
Reconciliation arbiter / orchestrator
e) During the deferred period - if any flows / group / meter
configurations are received via Inventory Config DTCN, arbiter shall push those
flows to current bundle
f) When timer expires, issue bundle closure OFP RPC call to all
switches in one close loop
Other key decisions :
1) Since we switch-agnostic implicit bundle inventory data model need not be
changed and that kind of change reuqires a more rigorous impact-assessment and
can be taken up in future
Regards,
Arun
From: [email protected]
[mailto:[email protected]] On Behalf Of Josh
Hershberg
Sent: Wednesday, February 28, 2018 1:26 PM
To: Anil Vishnoi <[email protected]>
Cc: Dayavanti Gopal Kamath <[email protected]>;
[email protected]
Subject: Re: [openflowplugin-dev] Approaches for making budle service more
generic (normal resync and upgrade resync being special cases)
A few comments inline...
On Tue, Feb 27, 2018 at 10:58 PM, Anil Vishnoi
<[email protected]<mailto:[email protected]>> wrote:
Hi Muthu,
Sorry for delayed response. Please see inline ..
On Fri, Feb 23, 2018 at 2:44 AM, Muthukumaran K
<[email protected]<mailto:[email protected]>> wrote:
Hi all,
@Arun, @Gobi, please feel free to add anything which I might have missed out
from our discussions
Context :
=======
During last OFPlugin weekly call, we were discussing about how bundle-based
resync flow would behave in upgrade context. In normal resync scenarios – ie.
a) switch reconnecting and/or b) controller reboots (and subsequent switch
reconnects), flows and groups which can be dispatched via resync bundle is
sourced from the inventory Config DS. In case of upgrade, inventory Config DS
would be completely nuked and hence a different handling is required. During
this discussion, a concern was raised as to why OFPlugin should become
extensively cognizant of “upgrade” as a special case. The thought process was
more to expose the bundle service generically from OFPlugin’s end so that
upgrade-orchestration becomes a higher level functionality without switching
behaviors from with OFP
So, following this mail boostraps the discussions with following approaches and
choosing an approach which is less invasive. We can continue to discuss the
merits, demerits and fine-tune the approach to arrive at an agreeable mechanism.
Premises :
=======
1) Bundle construct is currently used in Reconciliation and Resync context
2) The objective is to expose bundle as a generic construct for applications to
use whenever a set of flows and/groups have to be provisioned to the switch in
"atomic" manner
3) Usage scenarios
a) Common bundle shared across applications - usage example - coordinated
resync
b) Enabling applications to use bundles with specific set of constraints
if the merit of atomicity provided by bundles offsets the constraints (see
approach 2)
NOTE: Bundle is transactional concept, so to expose these generically we need
to make sure that the
API control loop is properly expose, so when user configure any bundle, it can
also fetch status of the bundle some way.
Bundle lifecycle :
===========
Approach 1 :
========
Tying up bundle permanently with a flow/group at DS level optionally. In other
words, a given flow is always associated with a fixed bundle-id. This can
tantamount to adding non-persistable (config = false) bundle-id attribute to
flow and group models. This appears to be a very rigid approach which would not
be flexible to associate flows and groups with different bundles for different
scenarios (normal operations and resync – to be specific)
This approach breaks the transactional nature of the bundle because now flows
will be tied to the bundle-id and it needs to be persisted till flow/groups
lifetime. Also if we really want to tag the flow/group we will have to user the
config data store, because from northbound side, operational data store can not
be modified. If we put in the config data store that means once FRM need to use
the same bundle-id, and that can be a major limitation because now application
also need to make sure that they are using the unique bundle-id. It can be done
in plugin as well, but point is somebody need to do this logistic handling and
that won't be simple.
Approach 2 :
========
Allowing applications to create and close the bundle with not more than one
bundle active per switch.
All groups and flows provisioned during the window of create and close of the
bundle by a given application can be pushed in context of the current-active
bundle. This approach has a benefit that flow / group object in DS need not
explicitly refer the bundle. But separate flavor of flow and group add/mod/del
API
needs to be invoked by apps to indicate that corresponding flows and group
actions have to be encapsulated within a bundle. It would be the responsibility
of
the application to close the bundle explictly
I think this make sense to me as a first cut implementation. Although we need
to provide programming interface (trough rpc, datastore) so that internal
and external application both can consume this functionality. Given that at any
point of time there is only active bundle for a switch, it should provide the
following interface to the applications
(1) Created/update/close bundle -- Already exist, but we need to
(2) Get bundles with the status (with current implementation it will return
only one bundle).
(3) Report proper error if application is trying to modify the bundle that is
already committed.
(4) notification/rpc for caching the last error message, because bundle can
throw error while staging as well and application should have some way to make
sure that the bundle they are staging is in error-free condition.
I like this option. One minor issue is, and I could be wrong about this, that
as far as I can tell the spec allows for multiple concurrent bundles per
switch. If this is true, then this design does not match the spec since it only
allows for a single bundle per switch at any given time. A possible mitigation
to this approach (which does not need to be implemented immediately) is to have
the notion of the "default bundle" which is what is described in Approach 2 and
in the future we could augment that with the ability to specify bundles both in
the CRUD/bundle management APIs that Anil mentioned and in the flow and croup
config writes. I don't think this is a major issue, just something that
occurred to me.
Approach 3 :
========
Keeping bundle(s) as floating, leasable, non-persistent resouce(s) which can be
associated with one or more flows/groups combination. Application functionality
- resync being one of them, can lease one or more bundle(s) as required from a
pool of bundles, associate the same with a switch and start using the same.
Some involved yang-modeling and service-logic would be required for providing
such leasing service and corresponding validation logic (eg. handling normal
releasing and abnormal releasing - lease-onwer node dies etc.) would require a
non-trivial service implementation and testing for various failure scenarios.
But this approach would be provide more flexibility from usage perspective.
In my humble opinion, this probably is not a good implementation given the
transactional nature of the bundles. I see bundle service as a platform feature
that user can leverage and it's should be pretty-much without any business
logic. Just like a normal finite state machine type API's where user create a
resource and take care of it's state during it's lifetime. Plugin should just
throw an error message whenever user try to do the operation that is not
allowed as per the openflowplugin specification, like using the already
committed bundle. So in this scenario, application is the one that need to
handle/coordinate the bundle lifetime, like the other resource flow/group. That
i think give more freedom to applications and it probably make the programming
model more simpler.
Approach 2 appears to be less invasive and is nothing but a special case of
Approach 3 (of course we can discuss on any other approaches and agree upon a
more acceptable way forward)
I would like to throw one more approach here, which probably can be more
simpler and have same control flow for most of the scenarios. Plugin should
provide a "reconcile" rpc that user can use to trigger reconciliation for
individual switch. So FRM won't do any reconciliation till application sends a
rpc request to trigger the reconciliation. This functionality will be enabled
when user enable the "orchestrator" profile in the openflowplugin config file
(we can choose any other name, it's just for discussion), otherwise everything
will work the way it works in the existing implementation.
This is how it should work in the two scenario you mentioned above
(1) Switch connection/disconnection : Configuration is already present in the
data store, so whenever application see the switch connects to the controller,
it can trigger the reconciliation. If bundle based reconciliation is enabled it
will use the bundle based reconciliation otherwise it will use the normal
reconciliation. In clustered environment, even if switch disconnect from one of
the controller, it won't notify the application about disconnection, so there
is no action required from the application. But if switch is disconnected from
all the controller, that will send notification to application and only "owner"
controller will react to the "reconcile" request.
(2) Controller reboot : This will work the same way as above. Only "owner"
controller will react to the "reconcile" request.
(3) Controller Upgrade : Same as above, but in this case user might want to
push all of the configuration to the switch and fire the reconcile request for
that switch. Rest works the same.
So from plugin perspective, it's very simple contract to the application --
Plugin will reconcile the configuration present in the config data store at the
time of reconcile request and then open it for general device configuration.
Although this approach does not give a control over the bundles, but it
provides a control to user on when to start the reconciliation.
This mechanism can be further enhanced when openflowplugin will expose bundle
to the application. Applications can use the bundles to reconcile in that
scenario, but this "reconcile" rpc will still be require to notify plugin to
open the gate for general configuration.
On a side note, we can think of using the data store as well to provide a
reconcile flag, but we need to discuss how we are going to maintain it's state
in data store for various event like switch connection/disconnection.
Let me know your thoughts.
So, we discussed this in some of the upgrade meetings and my opinion is that
it does not benefit the upgrade use case enough to justify the feature. Either
way we need a way for the application to control the bundles in the case
upgrade and however that needs to be trigger the upgrade installer/orchestrator
can trigger that before the switches reconnect. Also, note that even if we are
in reconciliation mode waiting to fill up the bundle, we still need to receive
the port status updates because we can't generate the flows without them. I
assume that shouldn't be too much of a problem.
Common model change across these approaches is following (assuming we choose
Approach 2)
=============================================================
1) A separate bunde-list to be added as child to flowcapablenode - to represent
generic 1:MANY relationship between switch and bundle for future extensions
2) Each element of bundle-list shall be a bundle object which has attributes -
bundle-id, bundle-status (OPEN|CLOSE), bundle-type (REGULAR for now and can
be added with additional enum values to provide one more level of
indirection in future if we want to add types like RESYNC - for example in
future)
3) Especially for supporting approach-2, there must not be more than 1 bundle
active for a DPN at any point in time. This should be imposed as a constraint in
the bundle-list. If we have more than 1 bundle active, then there must be
explicit mapping of flow and group entities to respective bundles
Or, as I suggest above, this can be postponed to a later stage of development
by introducing the addition convenient notion of a "default bundle".
I was wondering if we really want to maintain the bundle related data in
config data store? Given that bundle is transactional in nature and they does
not make sense across switch reconnect, controller reboot, i think providing
the rpc's for CRUD to maintain the lifecycle of bundle probably make more sense
here ? We can maintain the status of bundles in operational data store as well
because we will require that for clustering environment (to avoid routed rpc).
CRUD Handling of new bundle in bundle-list (assuming we choose Approach 2)
==================================================
1) Applications create a common shared bundle
(bundle-id=<bundle-id;bundle-status=OPEN;bundle-type=REGULAR> under bundle-list
and subsequently invoke special flowmod and groupmod APIs (to indicate that
corresponding flow-mods and group-mods are to be added in context of OPEN'ed
bundle)
By special means to add a leaf in the flow/group mod that we write to config
data store?
2) Bundle created under bundle-list can be disposed as part of nodal reboot
(since the bundle-list is operational in nature)
Yeah, I think using rpc here will help, because then we can just maintain the
status in operational data store and that's non-persistent anyways.
3) Bundle gets updated only for close status. ie. when applications decide to
close bundle which they have opened
4) Bundle is read mainly by OFPlugin via DTCN to get update events on bundle
OPENing and CLOSing (for initial case, this would be mainly for triggering
resync)
E2E flow (assuming Approach 2) :
======================
The proposal is to use
a) Reconciliation API which is present in OFPlugin to announce the
available of switch to apps
b) In reaction, apps OPEN a bundle using the approach 2 and push flows
and/or groups
c) When completed, application can CLOSE
Please see “app” referred above as normal resync or upgrade-specific resync
functionality and not in general notion of visualizing apps as ITM, IFM or
L3VPN service
Let us continue the discussion during next week’s call as well as over this
mail chain
Regards
Muthu
_______________________________________________
openflowplugin-dev mailing list
[email protected]<mailto:[email protected]>
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
--
Thanks
Anil
_______________________________________________
openflowplugin-dev mailing list
[email protected]<mailto:[email protected]>
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
_______________________________________________
openflowplugin-dev mailing list
[email protected]
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
