I agree, especially when the benefit of being able to use an existing service - yet only one which hasn't been deployed via a workflow - doesn't seem all that interesting IMO.
Another concern I could add to the ones you've mentioned is the service's inputs - the substituting template's inputs should be received via the properties of the abstract node in the top level service template. If the service already exists, these inputs would not be passed as expected. Ran On Wed, Aug 16, 2017 at 3:25 PM, D Jayachandran <d.jayachand...@ericsson.com > wrote: > Hi Ran, > > When Tal mentioned about "substituting service", I thought it was about > the services which dint have any associated executions/workflows triggered. > Am also in favor of "substituting service templates" rather than > "substituting service". > With "substituting service" approach (when the service is not > instantiated), I see some open points > - In a multi-user scenario, what will happen when a service is > composed using the substituting service and at the sametime a workflow is > triggered for the substituting service. ? > - Is it okay to delete(dissolve) the substituting service after it > is used to create the composed service. ? > > Starting with it might be a good idea to only have "substituting service > templates" approach. > > Regards, > DJ > -----Original Message----- > From: Ran Ziv [mailto:r...@cloudify.co] > Sent: Wednesday, August 16, 2017 4:29 PM > To: dev@ariatosca.incubator.apache.org > Subject: Re: Service Composition / Substitution Mapping > > I'd say right now we're looking at "static service composition" which is > only about "substituting service templates", not "substituting service". If > a service is already running, it will not be used. > > I think what Tal meant was that each service template - whether the top > level one or one of the substituting templates - needs to resolve its inner > reqs&caps internally first, and then resolve substitution reqs&caps across > service templates. > > > On Wed, Aug 16, 2017 at 12:00 PM, D Jayachandran < > d.jayachand...@ericsson.com> wrote: > > > Hi Tal, > > > > Thanks for organizing the points. > > So if I understand correctly we are looking only at "Static service > > composition" which includes "substituting service template" and > > "substituting service". > > > > As you said with "substituting service template" approach ,we will > > have all the nodes aggregated from other service templates and a > > single workflow would be triggered to perform life-cycle operation on > all the nodes. > > Am not sure why the workflows needs to be "boundary aware" for nodes > > being substituted ? I see nodes are already part of the composed > > service, Could you please help me understand this ? > > > > > > Regards, > > DJ > > -----Original Message----- > > From: Tal Liron [mailto:t...@cloudify.co] > > Sent: Saturday, August 12, 2017 4:52 AM > > To: dev@ariatosca.incubator.apache.org > > Subject: Re: Service Composition / Substitution Mapping > > > > You are correct -- to participate in this "multi-VIM" scenario, the > > Openstack plugin would have to know how to translate the TOSCA > > properties to a flavor ID. This could all be done in 100% TOSCA via > > policies (say, an aria.Openstack). > > > > Doing this automatically might not be a good idea, or even necessary. > > Worst case is you get a validation error if the ARIA plugin can't find > > a flavor in the table to match your requirements, in which you case > > you can go and manually find the right ID and add it to the table. > > > > And I agree about being fine with imperfection: the rule of thumb for > > our work is always to allow for sensible defaults even if no explicit > > policy is given. > > > > Anyway, we've gone way off the topic of this thread. We can talk about > > it more once it comes closer to an implementation. > > > > On Fri, Aug 11, 2017 at 3:52 PM, DeWayne Filppi <dewa...@cloudify.co> > > wrote: > > > > > Interesting. Take Openstack (please <rim shot>). If you model a > > > compute OS as explicitly as you like, there still has to be a "match" > > > to an Openstack image ID. Or are you saying you must supply the > > > image > > ID for > > > OSs. Likewise, you can't supply RAM and CPUs without a flavor ID. > > > Openstack does allow for custom flavors, but I doubt the plugin is > > > doing that. Much better to have a "caps-init" interface in some low > > > down base type that can be triggered at service creation to support > > > reqs/caps (IMHO). Then the plugin can verify whether the service > > > can be construct based on fuzzy constraints. Maybe this is a case > > > that the "general solution" is a nightmare of complexity, but having > > > a plugin scan the available flavors to make sure a requirement can > > > be met doesn't seem that tough. If TOSCA provided a formal > > > lifecycle interface for it, then orchestrators or just plugins could > > > determine how tricky they wanted to be. IOW, let not the perfect be > > > the enemy of > > the good. > > > > > > DeWayne > > > > > > > > > On Fri, Aug 11, 2017 at 1:26 PM, Tal Liron <t...@cloudify.co> wrote: > > > > > > > OK, that's a whole different can of worms. :) > > > > > > > > TOSCA's Compute capabilities (Container and OperatingSystem) are > > > explicit. > > > > You specify which OS you want, how much RAM you want, how many > > > > CPUs, > > etc. > > > > ARIA's explicit node types (for example, the AWS Compute node) are > > > likewise > > > > explicit. So there is not querying here: the plugin will attempt > > > > to spin > > > up > > > > exactly the virtual machine you asked for. If it fails, the > > > > workflow will fail. > > > > > > > > This is not good enough, I think, for real world scenarios. There > > > > are two possible solutions: > > > > > > > > 1) Support ranges or fallbacks. So instead of saying "I need 4 GB > > > > of > > RAM" > > > > you can say "I would like 4 GB of RAM, but 2 GB would also be OK." > > > There's > > > > no easy way to do this now without totally changing how the > > > > capability types are designed. But, it may be possible to override > > > > this via > > > policies. > > > > So, the capabilities would perhaps specify the minimal > > > > requirements, > > > while > > > > policies specify preferences. Some aspects of this were discussed > > > > in the OPEN-O project. DeWayne, has any of this survived in ONAP, > > > > or have we not reached that point in the discussion yet? > > > > > > > > 2) Incorporate this into the bigger topic of resource orchestration. > > > This a > > > > huge challenge for the industry. The problem field contains not > > > > just "I need X amount of RAM" but also "I want all my virtual > > > > machines and containers running on the same high-performance > > > > network backend and have these two nodes on the same bare-metal > > > > machine or at least in the same > > > data > > > > center rack with a NUMA interconnect, and I also don't want all > > > > this to cost more that $100 per hour." That's not crazy: these are > > > > real world requirements for high-performance VNFs and service > chaining. > > > > Resource orchestration requires a full map of what is available in > > > > the data > > > centers, > > > > a negotiation-based algorithm for properly allocating and placing > > > > resources, connection to billing services, etc. Of course resource > > > > orchestration is not within the scope of ARIA, but it would be > > > > great for ARIA to have plugins for them (and TOSCA be able to > > > > model resource requirement policies) when these become available. > > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 3:02 PM, DeWayne Filppi > > > > <dewa...@cloudify.co> > > > > wrote: > > > > > > > > > For the "resource realization" part, I was not even considering > > > > > multicloud/vim. I was considering single cloud even outside of > > > > > composition. Just reqs and caps. If my node "requires" a > > > > > compute node with Suse Linux version X with a minimum of 4GB > > > > > RAM, how does > > the > > > > > orchestrator match that without querying the cloud via the plugin? > > If > > > > it > > > > > does query it, which it seems it must in addition to the > > > > > implicit quota query, how is this done? TOSCA seems to not > > > > > really care, which is fine > > > > and > > > > > perhaps a good idea. But ARIA has to care. > > > > > > > > > > DeWayne > > > > > > > > > > On Fri, Aug 11, 2017 at 12:34 PM, Tal Liron <t...@cloudify.co> > wrote: > > > > > > > > > > > In my opinion, the new composite service should indeed be a > > > > > > single > > > > > service, > > > > > > so the ARIA CLI will show it as one (if a substituting service > > > already > > > > > > existed, it would be "dissolved" into the new top-level > > > > > > service). The composition will show its traces if you look > > > > > > more deeply, because > > > > you'll > > > > > > see that some node templates come from a different service > > template. > > > > > > Perhaps our CLI can detect this and tell the user which > > > > > > service > > > > templates > > > > > > were used to create the nodes in current service. > > > > > > > > > > > > The rest of what you described I think is not related directly > > > > > > to > > > > service > > > > > > composition, but resource realization. The way we currently > > > > > > handle > > > this > > > > > is > > > > > > by being very explicit: you have to use the derived Compute > > > > > > types > > > > > included > > > > > > in our AWS or Openstack plugins, for example. > > > > > > > > > > > > I think we should still keep this, because sometimes you want > > > > > > to be > > > > very > > > > > > explicit (and use specific features of the platform), but I > > > > > > actually > > > > have > > > > > > some ideas for multi-VIM support that would work differently: > > > > > > the > > > idea > > > > is > > > > > > to use the basic Compute type, with detailed properties in its > > > > Container > > > > > > and OperatingSystem capabilities. There would then be a > > > > > > policy, > > > perhaps > > > > > > aria.Realization, that would hint at what kind of Compute > > > > > > nodes you > > > are > > > > > > looking for (you can apply the policy to specific nodes or > > > > > > groups, or generally). It might be possible to still have > > > > > > strong > > > platform-specific > > > > > > feature support here, perhaps by implementing a general string > > > > key-value > > > > > > dict that could include hints for the specific plugin to use. > > > > > > > > > > > > Anyway, just ideas at this point... > > > > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 2:25 PM, DeWayne Filppi > > > > > > <dewa...@cloudify.co > > > > > > > > > > wrote: > > > > > > > > > > > > > Good stuff. Obviously a "fancy include" was the wrong > metaphor. > > > The > > > > > > > lifecycles are linked though. When a create the referencing > > > service > > > > > > (aria > > > > > > > service create), ARIA will match the other service template, > > > > > > > and > > > run > > > > > > create > > > > > > > service on it, and then continue on. Uninstall of the > > > > > > > referencer > > > > will > > > > > > > destroy the referenced service. After such an > > > > > > > instantiation, would > > > > > > listing > > > > > > > all services produce two (perhaps with some parent > > > > > > > indicator) or > > > one? > > > > > > From > > > > > > > what I've seen, reqs and caps don't have the equivalent of a > > > > lifecycle > > > > > > > interface associated. Maybe I missed it. It is implicit in > > > > > > > the orchestrator to match stuff up at service creation time, > > > > > > > I'm > > > > assuming, > > > > > > and > > > > > > > using plugins somehow? The sort of flagship scenario is > > > > > > > matching > > > > cloud > > > > > > > compute resources without specifying an image. Since > > > > > > > discovering > > > > > > whether a > > > > > > > particular cloud has a given capability, I guessing a plugin > > > > > > > for > > > that > > > > > > cloud > > > > > > > has to search cloud inventory and perhaps quota to provide > > > > > > > info for > > > > the > > > > > > > match. Since plugins (at least the Cloudify variety) tend > > > > > > > to be > > > > > > triggered > > > > > > > due to operations that they connect to lifecycle methods, > > > > > > > and there > > > > is > > > > > no > > > > > > > "materialize_capabilities" interface that I noticed, some > > > > > > > other > > > kind > > > > of > > > > > > > magic must be performed. Maybe this is an > > > > > > > orchestrator-specific > > > > detail > > > > > > > outside of TOSCA. > > > > > > > > > > > > > > DeWayne > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 10:57 AM, Tal Liron > > > > > > > <t...@cloudify.co> > > > wrote: > > > > > > > > > > > > > > > OK, let me try to organize this differently. Three > > > > > > > > potential > > > > > > conceptions > > > > > > > > here: > > > > > > > > > > > > > > > > 1) A "fancy include," as you say. All that would happen > > > > > > > > here is > > > > that > > > > > > the > > > > > > > > TOSCA parser would automatically find the service template > > > > > > > > to > > > > include > > > > > > > from > > > > > > > > some kind of repository of recognized service templates, > > > > > > > > and just > > > > > > include > > > > > > > > that. The language in the TOSCA spec suggests that this is > > > > > > > > not > > > the > > > > > > > > intention: it is something that happens at the > "orchestrator," > > > not > > > > > the > > > > > > > > "parser." > > > > > > > > > > > > > > > > 2) Static service composition. This happens not at the > > > > > > > > parsing > > > > stage, > > > > > > but > > > > > > > > rather the instantiation stage, where reqs-and-caps happens. > > > > > > > > I > > > > think > > > > > > this > > > > > > > > is what is intended: substitution mapping is specifically > > > > > > > > about > > > > > mapping > > > > > > > > reqs-and-caps. And this is also where things like > > > > > > > > scalability and > > > > > > > placement > > > > > > > > happen: think of a requirement matching a capability, but > > > > > > > > that > > > > > > capability > > > > > > > > not having enough capacity. So, the node might be able to > > > > > > > > scale > > > > out: > > > > > in > > > > > > > the > > > > > > > > case of substitution, this would mean duplicating an > > > > > > > > entire > > > service > > > > > > > > instance. My understanding is that this is what is > > > > > > > > intended by > > > > TOSCA, > > > > > > and > > > > > > > > it's entirely within the scope of what we can do. We've > > > > > > > > recently > > > > just > > > > > > > > refactored the instantiation phase into a new "topology" > > > > > > > > module > > > > where > > > > > > > > exactly all this logic is concentrated. So think of it > > > > > > > > this way > > > -- > > > > > ARIA > > > > > > > has > > > > > > > > three parts: "parser," "topology manager," and "workflow > > engine" > > > > > > > > ("orchestrator"). > > > > > > > > > > > > > > > > (I think I might have confused some thing here a bit when > > > > mentioning > > > > > a > > > > > > > > "logical proxy node." I do not mean an actual piece of > > > > > > > > proxy > > > > software > > > > > > > > running on a machine somewhere. I mean just a data point > > > > > > > > in the ARIA-generated topology that can be used as a > > > > > > > > barrier of sorts > > > when > > > > > > > > constructing the task graph -- because the task graph > > > > > > > > follows relationships, the edges of the topology. It could > > > > > > > > be that we > > > > > discover > > > > > > in > > > > > > > > our POC that this is not needed, because actually the > > > substitution > > > > > node > > > > > > > is > > > > > > > > already part of our topology data model and we might be > > > > > > > > able to > > > > > easily > > > > > > > take > > > > > > > > that into account when generating the task graph.) > > > > > > > > > > > > > > > > 3) Live service composition. I think this is a fantasy of > > > > > > > > some > > > > > people: > > > > > > > that > > > > > > > > ARIA would be able to take existing, "running" service > > > > > > > > chains and > > > > run > > > > > > > > workflows with them. I do think this is a solvable > > > > > > > > problem, but > > > not > > > > > via > > > > > > > > substitution mapping per se. A solution could involve > > > > > > > > deployment > > > > of a > > > > > > > proxy > > > > > > > > service (which could actually be encapsulated in a > > > > > > > > substitution > > > > > > mapping, > > > > > > > > but doesn't have to be), or configuring specialized > > > > > > > > virtual ports > > > > via > > > > > > an > > > > > > > > SDN controller, or via logical proxy nodes created via an > > > > inspection > > > > > > > tool, > > > > > > > > etc. I cannot believe that there is a one-size-fits-all > > > > > > > > solution > > > to > > > > > > this > > > > > > > > problem. The dream of NFV might be to have everything > > > > > > > > connect to > > > > each > > > > > > > other > > > > > > > > like LEGO blocks, but there are far too many protocols, > > > > > configuration, > > > > > > > and > > > > > > > > security standards. I think point #2, what I called > > > > > > > > "static > > > service > > > > > > > > composition," is a realistic compromise and within the > > > > > > > > scope of > > > > what > > > > > > > TOSCA > > > > > > > > promises. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 12:08 PM, DeWayne Filppi < > > > > > dewa...@cloudify.co> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > To my eyes, the spec doesn't speak of runtime service > > > > substituion, > > > > > > but > > > > > > > > > parsetime template composition. IOW, substitution > > > > > > > > > mapping is a > > > > > fancy > > > > > > > > > "include", or the equivalent of an interface definition. > > > > > > > > > Is it > > > > > > > > understood > > > > > > > > > by the ARIA team that this includes proxying of running > > > services? > > > > > > IOW, > > > > > > > > if > > > > > > > > > my template requires a database service that my template > > > > > > > > > does > > > > *not* > > > > > > > want > > > > > > > > to > > > > > > > > > control the lifecycle of, I can "substitution map" an > > > > > > > > > instance > > > > of a > > > > > > > > > template (i.e. a running service)? This would be a > > > > > > > > > lovely > > > > feature, > > > > > > but > > > > > > > > > it's not really a "substitution map", rather more of a > > > > > > > > > "service > > > > > > proxy" > > > > > > > > (as > > > > > > > > > implemented as a plugin in Cloudify). Just trying to > > clarify. > > > > > > Maybe > > > > > > > > the > > > > > > > > > community thinks that "substitution map" as something > > > > > > > > > that > > > occurs > > > > > > > beyond > > > > > > > > > parsetime, or should. > > > > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 9:52 AM, Tal Liron > > > > > > > > > <t...@cloudify.co> > > > > > wrote: > > > > > > > > > > > > > > > > > > > Well, DJ, it's all just opinion at this stage and > > > > > > > > > > feedback is > > > > > > > welcome! > > > > > > > > > > > > > > > > > > > > Option 1: > > > > > > > > > > > To look at satisfying nodes present in a > > > substituting > > > > > > > > service, > > > > > > > > > > > Have these nodes part of the newly created service > > > > > > > > > > > and > > > remove > > > > > the > > > > > > > > > > > substituting service(nodes with different ID's. Also > > > > > > > > > > > we are > > > > > very > > > > > > > much > > > > > > > > > in > > > > > > > > > > > favor of UUID ) > > > > > > > > > > > With this approach I guess the substituting > > > > > > > > > > > service > > > > > > should > > > > > > > > not > > > > > > > > > > > have any associated workflows running. If at all an > > > workflow > > > > > > > > execution > > > > > > > > > is > > > > > > > > > > > already triggered I hope this service will not be > > > considered > > > > > for > > > > > > > > > > > substitution. > > > > > > > > > > > I hope this is the correct approach when we > > > > > > > > > > > are > > > > looking > > > > > > at > > > > > > > a > > > > > > > > > > > service for the substitution > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, this is a good idea. It would be easy to discover > > > > > > > > > > this > > > > > > according > > > > > > > > to > > > > > > > > > > the stored node state -- just make sure that all nodes > > > > > > > > > > are > > > in a > > > > > > > stable > > > > > > > > > > state before composition. > > > > > > > > > > > > > > > > > > > > This leads to a general issue: before composition, the > > > > > substituting > > > > > > > > > > services must be validated in some way before > > > > > > > > > > composition > > > > begins. > > > > > > > > > > > > > > > > > > > > Also, let's all start using TOSCA terminology here: > > > > > > > > > > the > > > > > containing > > > > > > > > > service > > > > > > > > > > is called the "top-level service," and the contained > > > > > > > > > > services > > > > are > > > > > > > > called > > > > > > > > > > "substituting services." > > > > > > > > > > > > > > > > > > > > Also, we keep using trivial examples, but it's > > > > > > > > > > definitely > > > > > possible > > > > > > > for > > > > > > > > a > > > > > > > > > > top-level service to require several substituting > > > > > > > > > > services at > > > > the > > > > > > > same > > > > > > > > > > time. I can definitely see such things happening in > > > > > > > > > > NFV with > > > > even > > > > > > > > simple > > > > > > > > > > service chains. Basically every VNF could be a > > > > > > > > > > substituting > > > > > > service. > > > > > > > > > > > > > > > > > > > > So, actually one of the validations would be to make > > > > > > > > > > sure you > > > > do > > > > > > not > > > > > > > > > create > > > > > > > > > > circular composition: if the top-level also has > > > > > > > > > > substitution > > > > > > > mappings, > > > > > > > > > you > > > > > > > > > > need to make sure that one of the substituting ones > > > > > > > > > > doesn't > > > > > require > > > > > > > it. > > > > > > > > > :) > > > > > > > > > > Not a very likely situation, but it would lead to > failure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Option 2: > > > > > > > > > > > While creating a service look at the > > > > > > > > > > > req-caps at > > > the > > > > > > > > > > > service-template level and create a service > > > > > > > > > > > including the > > > > nodes > > > > > > > > > provided > > > > > > > > > > by > > > > > > > > > > > the substituting service-template. With this > > > > > > > > > > > approach there > > > > > would > > > > > > > not > > > > > > > > > be > > > > > > > > > > > any service created from the service-template which > is > > > > > > providing > > > > > > > > the > > > > > > > > > > > substitution functionality. The service-template > > > > > > > > > > > would > > > remain > > > > > the > > > > > > > > same > > > > > > > > > > but > > > > > > > > > > > only the service would be added with extra nodes. > > > > > > > > > > > > > > > > > > > > > > Are you considering both option 1 & 2 for the > > > implementation > > > > ? > > > > > If > > > > > > > not > > > > > > > > > > > which one do you feel which take priority. I see > > > > > > > > > > > option 2 > > > at > > > > > this > > > > > > > > stage > > > > > > > > > > > could be the best possible approach Also could you > > > > > > > > > > > please let me know a tentative time for this > > > > > > feature > > > > > > > > to > > > > > > > > > be > > > > > > > > > > > available? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think both options 1 and 2 make sense and are > > > > > > > > > > useful, and > > > > > > actually > > > > > > > > one > > > > > > > > > is > > > > > > > > > > a subset of the other. > > > > > > > > > > > > > > > > > > > > With option #2 (substituting a service template), it > > > > > > > > > > means > > > new > > > > > > nodes > > > > > > > > are > > > > > > > > > > instantiated and the composed service would include > > > > > > > > > > all > > > nodes. > > > > > So, > > > > > > an > > > > > > > > > > "install" workflow would install everything at once. > > > > > > > > > > In this > > > > case > > > > > > we > > > > > > > do > > > > > > > > > > need to fix the lifecycle workflows to be "boundary > aware," > > > so > > > > > that > > > > > > > > > > workflows of substituting service nodes are part of > > > > > > > > > > their own > > > > > task > > > > > > > > > graph. I > > > > > > > > > > think that possibly using a logical proxy node in > > > > > > > > > > between > > > might > > > > > > solve > > > > > > > > > this > > > > > > > > > > situation automatically. > > > > > > > > > > > > > > > > > > > > With option #1 (substituting a service) the > > > > > > > > > > substituting > > > nodes > > > > > > might > > > > > > > > > > already be installed. Or not (they might have been > > > instantiated > > > > > but > > > > > > > > still > > > > > > > > > > not installed). So, the lifecycle workflows should > > > > > > > > > > only work > > > on > > > > > the > > > > > > > > nodes > > > > > > > > > > that have not yet been installed. > > > > > > > > > > > > > > > > > > > > The point is that we just need to beef up the > > > > > > > > > > lifecycle > > > > workflows > > > > > > to > > > > > > > > > > properly work with boundaries and with a mix of nodes > > > > > > > > > > in > > > > > different > > > > > > > > > states. > > > > > > > > > > If they can do that, then they can handle any kind of > > > > > > > > > > service > > > > > > > > > composition, > > > > > > > > > > whether it's option #1 or option #2. > > > > > > > > > > > > > > > > > > > > I don't think we can provide a timeline yet, but I > > > > > > > > > > will say > > > > that > > > > > we > > > > > > > are > > > > > > > > > in > > > > > > > > > > the research phase and may have a POC soon. Avia is in > > > > > > > > > > charge > > > > of > > > > > > this > > > > > > > > > JIRA, > > > > > > > > > > so I will let him chime in with the current state of > > things. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
