On Mon, Mar 12, 2018 at 10:51 PM, D Arunprakash <[email protected]> wrote:
> Last week’s discussion on reconciliation/upgrade profiles, > > > > 1. There shall be two profiles/modes configured which can change OFP > behavior in context of reconciliation > > Reconciliation Mode : <ARBITARTED|AUTONOMOUS> > > AUTONOMOUS mode implies that OFPlugin shall perform reconciliation > autonomously as it does now without any change in the workflow - ie. > Switch- > connection-handling followed by bundle-based Reconciliaion execution > followed by publishing of switch to the Inventory Opertaional DS > > ARBITRATED mode implies that OFPlugin shall not execute reconciliation > by itself but by allowing higher level app "Reconciliation arbiter / > orchestrator" to initiate and > complete the bundle-based reconciliation including any error-handling > thereof > This sounds good, but w e need to take care of normal reconciliation as well here, because Arbitrated mode can be used OVS as well as Hardware based switch. So Orchestrator will just call reconcile, but how the reconciliation is done is depends on whether plugin configured the switch for bundle based reconciliation or normal reconciliation. > > > 2. This implies two changes for OFPlugin > a) OFP must publish the switch to Inventory Operational soon after the > connection-handling and master-election phases complete without triggering > Reconciliation by > itself in ARBITRATED mode > Do we want to collect stats before publishing the node. Even staging the bundle can throw errors, where stats can be handy for making the decision. Also to handle normal reconciliation, it might require the stats to properly fire group/flow mod. > > b) OFP must expose RPCs to start, modify and close bundle so that > "Reconciliation arbiter / orchestrator" is able to invoke the same to > realize reconciliation > This is the final goal or we are planning to implement it for the first cut implementation as well? If i remember we were thinking of using a default internal bundle for reconciliation and later we will open APIs for bundle management to the user as well. (c) Proper error handling and protection against these mode, e.g if plugin is in Autonomous mode, user firing the reconciliation rpc will be returned with error. > > > Reconciliation Arbiter / Orchestrator (which is functional only in > scenario when Reconciliation-Mode==ARBITRATED) > ============================================================ > ================ > Listeners for Reconciliation Arbiter / Orchestrator : > This is more or less similar to current reconciliation function of FRM > a) Inventory Oper DTCN for switch status sensing > b) Config Inventory DTCN for flows / groups and meters > > Conditions: > ------------ > a) Whether or not data is available for given switch in Config Inventory DS > b) Whether the system is in upgrade mode > > Common reaction for Switch Connection handling when > - Condition 1 : UPGRADE_MODE=true > - Condition 2 : UPGRADE_MODE=false > For all conditions, reconciliation can be executed by the arbiter as per > below steps > I believe this configuration is in Orchestrator side ? > > > Trigger reconciliation by > a) Issuing implicit close of implicit bundle-id - this is mainly > to address following failure scenario in OF-HA context : For OFHA purge the > incomplete implicit bundles > which might have been started by previous switch-master in > case leadership changes. Since we use a single implicit bundle-id, issuing > a cancel-bundle from new > switch-master implicitly would not result in any issues and > can be considered as safety measure. > b) starting implicit bundle using OFP RPC > b) reading flows / groups / meters from DS if present > c) add the same to the implicit bundle if available > d) defer the bundle close for a period of time (which is not a > timer managed by OFP but by some static configuration managed within the > scope of > Reconciliation arbiter / orchestrator > I think it's good to have this time in the plugin as well. We can set it to 0, if user want to wait for the bundle to wait forever. If Orchestrator want to control the timeout, they can set OFP flag to 0. > > e) During the deferred period - if any flows / group / meter > configurations are received via Inventory Config DTCN, arbiter shall push > those flows to current bundle > f) When timer expires, issue bundle closure OFP RPC call to all > switches in one close loop > Rest looks good. > > > Other key decisions : > 1) Since we switch-agnostic implicit bundle inventory data model need not > be changed and that kind of change reuqires a more rigorous > impact-assessment and can be taken up in future > > > I believe two things are missing here (1) How plugin will know when to open a implicit bundle for reconciliation? (2) How plugin will know when to close the bundle ? If we are assuming that UPGRADE_MODE is the global flag that will define when the reconciliation of the whole network should start and stop, this probably is okay first approach. But i was thinking of exposing RPC/DataStore way from OFP, so that user can call those interface to notify plugin to start the reconciliation (open bundle) of the specific device and when to commit it (close bundle). This can give more control to orchestrator and plugin don't need to handle the global state across all the devices connected to it (which can keep changing over the time). > > Regards, > > Arun > > > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Josh > Hershberg > *Sent:* Wednesday, February 28, 2018 1:26 PM > *To:* Anil Vishnoi <[email protected]> > *Cc:* Dayavanti Gopal Kamath <[email protected]>; > [email protected] > *Subject:* Re: [openflowplugin-dev] Approaches for making budle service > more generic (normal resync and upgrade resync being special cases) > > > > A few comments inline... > > > > On Tue, Feb 27, 2018 at 10:58 PM, Anil Vishnoi <[email protected]> > wrote: > > Hi Muthu, > > > > Sorry for delayed response. Please see inline .. > > > > On Fri, Feb 23, 2018 at 2:44 AM, Muthukumaran K < > [email protected]> wrote: > > Hi all, > > > > @Arun, @Gobi, please feel free to add anything which I might have missed > out from our discussions > > > > Context : > > ======= > > During last OFPlugin weekly call, we were discussing about how > bundle-based resync flow would behave in upgrade context. In normal resync > scenarios – ie. a) switch reconnecting and/or b) controller reboots (and > subsequent switch reconnects), flows and groups which can be dispatched via > resync bundle is sourced from the inventory Config DS. In case of upgrade, > inventory Config DS would be completely nuked and hence a different > handling is required. During this discussion, a concern was raised as to > why OFPlugin should become extensively cognizant of “upgrade” as a special > case. The thought process was more to expose the bundle service generically > from OFPlugin’s end so that upgrade-orchestration becomes a higher level > functionality without switching behaviors from with OFP > > > > So, following this mail boostraps the discussions with following > approaches and choosing an approach which is less invasive. We can continue > to discuss the merits, demerits and fine-tune the approach to arrive at an > agreeable mechanism. > > > > Premises : > > ======= > 1) Bundle construct is currently used in Reconciliation and Resync context > 2) The objective is to expose bundle as a generic construct for > applications to use whenever a set of flows and/groups have to be > provisioned to the switch in "atomic" manner > 3) Usage scenarios > a) Common bundle shared across applications - usage example - > coordinated resync > b) Enabling applications to use bundles with specific set of > constraints if the merit of atomicity provided by bundles offsets the > constraints (see approach 2) > > NOTE: Bundle is transactional concept, so to expose these generically we > need to make sure that the > > > > API control loop is properly expose, so when user configure any bundle, > it can also fetch status of the bundle some way. > > > > Bundle lifecycle : > > =========== > Approach 1 : > > ======== > Tying up bundle permanently with a flow/group at DS level optionally. In > other words, a given flow is always associated with a fixed bundle-id. This > can > > > > tantamount to adding non-persistable (config = false) bundle-id attribute > to flow and group models. This appears to be a very rigid approach which > would not be flexible to associate flows and groups with different bundles > for different scenarios (normal operations and resync – to be specific) > > This approach breaks the transactional nature of the bundle because now > flows will be tied to the bundle-id and it needs to be persisted till > flow/groups lifetime. Also if we really want to tag the flow/group we will > have to user the config data store, because from northbound side, > operational data store can not be modified. If we put in the config data > store that means once FRM need to use the same bundle-id, and that can be a > major limitation because now application also need to make sure that they > are using the unique bundle-id. It can be done in plugin as well, but point > is somebody need to do this logistic handling and that won't be simple. > > > Approach 2 : > > ======== > Allowing applications to create and close the bundle with not more than > one bundle active per switch. > All groups and flows provisioned during the window of create and close of > the bundle by a given application can be pushed in context of the > current-active > bundle. This approach has a benefit that flow / group object in DS need > not explicitly refer the bundle. But separate flavor of flow and group > add/mod/del API > needs to be invoked by apps to indicate that corresponding flows and group > actions have to be encapsulated within a bundle. It would be the > responsibility of > the application to close the bundle explictly > > I think this make sense to me as a first cut implementation. Although we > need to provide programming interface (trough rpc, datastore) so that > internal and external application both can consume this functionality. > Given that at any point of time there is only active bundle for a switch, > it should provide the following interface to the applications > > > > (1) Created/update/close bundle -- Already exist, but we need to > > (2) Get bundles with the status (with current implementation it will > return only one bundle). > > (3) Report proper error if application is trying to modify the bundle > that is already committed. > > (4) notification/rpc for caching the last error message, because bundle > can throw error while staging as well and application should have some way > to make sure that the bundle they are staging is in error-free condition. > > > > I like this option. One minor issue is, and I could be wrong about this, > that as far as I can tell the spec allows for multiple concurrent bundles > per switch. If this is true, then this design does not match the spec since > it only allows for a single bundle per switch at any given time. A possible > mitigation to this approach (which does not need to be implemented > immediately) is to have the notion of the "default bundle" which is what is > described in Approach 2 and in the future we could augment that with the > ability to specify bundles both in the CRUD/bundle management APIs that > Anil mentioned and in the flow and croup config writes. I don't think this > is a major issue, just something that occurred to me. > > > > > > > > > Approach 3 : > > ======== > Keeping bundle(s) as floating, leasable, non-persistent resouce(s) which > can be associated with one or more flows/groups combination. Application > functionality - resync being one of them, can lease one or more bundle(s) > as required from a pool of bundles, associate the same with a switch and > start using the same. Some involved yang-modeling and service-logic would > be required for providing such leasing service and corresponding validation > logic (eg. handling normal releasing and abnormal releasing - lease-onwer > node dies etc.) would require a non-trivial service implementation and > testing for various failure scenarios. But this approach would be provide > more flexibility from usage perspective. > > In my humble opinion, this probably is not a good implementation given > the transactional nature of the bundles. I see bundle service as a platform > feature that user can leverage and it's should be pretty-much without any > business logic. Just like a normal finite state machine type API's where > user create a resource and take care of it's state during it's lifetime. > Plugin should just throw an error message whenever user try to do the > operation that is not allowed as per the openflowplugin specification, like > using the already committed bundle. So in this scenario, application is the > one that need to handle/coordinate the bundle lifetime, like the other > resource flow/group. That i think give more freedom to applications and it > probably make the programming model more simpler. > > > > Approach 2 appears to be less invasive and is nothing but a special case > of Approach 3 (of course we can discuss on any other approaches and agree > upon a more acceptable way forward) > > > > I would like to throw one more approach here, which probably can be more > simpler and have same control flow for most of the scenarios. Plugin should > provide a "reconcile" rpc that user can use to trigger reconciliation for > individual switch. So FRM won't do any reconciliation till application > sends a rpc request to trigger the reconciliation. This functionality will > be enabled when user enable the "orchestrator" profile in the > openflowplugin config file (we can choose any other name, it's just for > discussion), otherwise everything will work the way it works in the > existing implementation. > > > > This is how it should work in the two scenario you mentioned above > > (1) Switch connection/disconnection : Configuration is already present in > the data store, so whenever application see the switch connects to the > controller, it can trigger the reconciliation. If bundle based > reconciliation is enabled it will use the bundle based reconciliation > otherwise it will use the normal reconciliation. In clustered environment, > even if switch disconnect from one of the controller, it won't notify the > application about disconnection, so there is no action required from the > application. But if switch is disconnected from all the controller, that > will send notification to application and only "owner" controller will > react to the "reconcile" request. > > > > (2) Controller reboot : This will work the same way as above. Only "owner" > controller will react to the "reconcile" request. > > > > (3) Controller Upgrade : Same as above, but in this case user might want > to push all of the configuration to the switch and fire the reconcile > request for that switch. Rest works the same. > > > > So from plugin perspective, it's very simple contract to the application > -- Plugin will reconcile the configuration present in the config data store > at the time of reconcile request and then open it for general device > configuration. Although this approach does not give a control over the > bundles, but it provides a control to user on when to start the > reconciliation. > > > > This mechanism can be further enhanced when openflowplugin will expose > bundle to the application. Applications can use the bundles to reconcile in > that scenario, but this "reconcile" rpc will still be require to notify > plugin to open the gate for general configuration. > > > > On a side note, we can think of using the data store as well to provide a > reconcile flag, but we need to discuss how we are going to maintain it's > state in data store for various event like switch > connection/disconnection. > > > > Let me know your thoughts. > > > > So, we discussed this in some of the upgrade meetings and my opinion is > that it does not benefit the upgrade use case enough to justify the > feature. Either way we need a way for the application to control the > bundles in the case upgrade and however that needs to be trigger the > upgrade installer/orchestrator can trigger that before the switches > reconnect. Also, note that even if we are in reconciliation mode waiting to > fill up the bundle, we still need to receive the port status updates > because we can't generate the flows without them. I assume that shouldn't > be too much of a problem. > > > > > > > > > Common model change across these approaches is following (assuming we > choose Approach 2) > > ============================================================= > 1) A separate bunde-list to be added as child to flowcapablenode - to > represent generic 1:MANY relationship between switch and bundle for future > extensions > 2) Each element of bundle-list shall be a bundle object which has > attributes - bundle-id, bundle-status (OPEN|CLOSE), bundle-type (REGULAR > for now and can > be added with additional enum values to provide one more level of > indirection in future if we want to add types like RESYNC - for example in > future) > 3) Especially for supporting approach-2, there must not be more than 1 > bundle active for a DPN at any point in time. This should be imposed as a > constraint in > the bundle-list. If we have more than 1 bundle active, then there > must be explicit mapping of flow and group entities to respective bundles > > > > Or, as I suggest above, this can be postponed to a later stage of > development by introducing the addition convenient notion of a "default > bundle". > > > > > > > > > > I was wondering if we really want to maintain the bundle related data in > config data store? Given that bundle is transactional in nature and they > does not make sense across switch reconnect, controller reboot, i think > providing the rpc's for CRUD to maintain the lifecycle of bundle probably > make more sense here ? We can maintain the status of bundles in operational > data store as well because we will require that for clustering environment > (to avoid routed rpc). > > > CRUD Handling of new bundle in bundle-list (assuming we choose Approach 2) > > ================================================== > 1) Applications create a common shared bundle (bundle-id=<bundle-id;bundle- > status=OPEN;bundle-type=REGULAR> under bundle-list and subsequently > invoke special flowmod and groupmod APIs (to indicate that corresponding > flow-mods and group-mods are to be added in context of OPEN'ed bundle) > > By special means to add a leaf in the flow/group mod that we write to > config data store? > > > > 2) Bundle created under bundle-list can be disposed as part of nodal > reboot (since the bundle-list is operational in nature) > > Yeah, I think using rpc here will help, because then we can just > maintain the status in operational data store and that's non-persistent > anyways. > > 3) Bundle gets updated only for close status. ie. when applications decide > to close bundle which they have opened > 4) Bundle is read mainly by OFPlugin via DTCN to get update events on > bundle OPENing and CLOSing (for initial case, this would be mainly for > triggering resync) > > > > E2E flow (assuming Approach 2) : > > ====================== > > The proposal is to use > > a) Reconciliation API which is present in OFPlugin to announce the > available of switch to apps > > b) In reaction, apps OPEN a bundle using the approach 2 and push > flows and/or groups > > c) When completed, application can CLOSE > > > > Please see “app” referred above as normal resync or upgrade-specific > resync functionality and not in general notion of visualizing apps as ITM, > IFM or L3VPN service > > > > Let us continue the discussion during next week’s call as well as over > this mail chain > > > > Regards > > Muthu > > > > > > > > > _______________________________________________ > openflowplugin-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev > > > > > > -- > > Thanks > > Anil > > > _______________________________________________ > openflowplugin-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev > > > -- Thanks Anil
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
