I don't think the implementation field is an input at all. It is instead the thing that receives the inputs. :)
In TOSCA, the implementation is actually one primary artifact plus optional dependencies. It should receive the interface and operation inputs, but nothing else. TOSCA has nothing to say about "tasks", and indeed that's an implementation (ha!) detail of ARIA. In any case, all of this is indeed gone when we move to ARIA tasks: tasks only have "function" and "arguments", none of the other fields. When we have an artifact management system, that "implementation"+"dependencies" field will become much more important: we will validate that the artifact exists in the CSAR, see about copying it remotely, support the get_artifact intrinsic function, etc... On Thu, May 25, 2017 at 4:12 AM, Maxim Orlov <[email protected]> wrote: > Not entirely sure why the "implementation" gets its own field. It basically > another input to the operation. Though it's a special input (since it > enables for the plugin do decide which "function" to run), it's still an > inputs. feel we already have a lot of different possible inputs to an > operation: (1) interface inputs, (2) operation inputs (3) task inputs. I > feel that the "implementation" should have a special place under the > operation inputs, but not a field on its own... > > On Thu, May 25, 2017 at 1:22 AM, Tal Liron <[email protected]> wrote: > > > We had a long face-to-face today and reached an alignment. ARIA-149 is > > being updated to reflect it. > > > > So, we are going all the way with this: > > > > The "implementation" and "inputs" fields in the models refer *only* to > what > > comes from the TOSCA parser. They are passed from template to instance as > > is. This means that "implementation" should always be considered an > > artifact name. (Indeed, when we implement artifact in the near future, > this > > field may be refactored in some way, possibly to become a foreign key to > an > > artifact-related model.) > > > > The OperationTemplate and Operation models both also have a > "configuration" > > field, a dict of special parameters used as hints for orchestration. The > > parser can generate them using our magical "dependencies" grammar > > extensions. They are normal parameters (typed, and can contain intrinsic > > functions). > > > > And, we are introducing two *new* fields to Operation and Task models > (and > > the related Task API): "function" and "arguments". "function" is the full > > path to a Python function, and "arguments" is a dict of parameters that > are > > sent as arguments to the function. These are ARIA's implementation > details > > for orchestration, and are *not* directly related to TOSCA's > > "implementation" and "inputs" fields, and indeed the Task model does not > > have nor need the "implementation" and "inputs" (to put it another way: > > we've renamed these two fields in Task, but they have the exact same > usage > > as before). > > > > The logic for tasks has not changed, but let's go over it again for > > clarification with the new fields in mind: > > > > 1. The parser generates an OperationTemplate model, and sets the > > "implementation" and "inputs" fields. Two ARIA grammar extensions > > optionally apply: > > 1. If the ">" is used in the "implementation", then the parser > actually > > leaves "implementation" empty (there is no artifact) and instead > sets > > "function". > > 2. If the ">" is used in "dependencies", they are converted to > > "configuration". > > 2. During instantiation, all these fields are copied as is from > > OperationTemplate to Operation. > > 3. During the configuration phase of instantiation, there are two > > possible routes: > > 1. For the default execution plugin, it generates "function" and > > "arguments" according to whether the operation is local or remote, > > using > > params from "configuration" if they are available (and valid). > Also, > > any > > extra "configuration" params set by the user, which are not > > recognized by > > the plugin, just get appended to "arguments". > > 2. For other plugins, we already have "function" set for us by the > > parser (according to what the user put in "implementation" after > > the ">"). > > So, we just append all "configuration" params if they exist to > > "arguments" > > 4. For all plugins, "inputs" are also appended to "arguments". > > > > Some implications of this logic worth considering: > > > > 1. "implementation" and "inputs" are never mangled and always adhere > to > > the strict TOSCA typed interface contract. > > 2. It's possible to use our special "dependencies" grammar to send > > *extra > > arbitrary* arguments to the Python operation function, *beyond* what > is > > allowed by the strict TOSCA contract. > > 3. Python operation functions do not have a way to know which > arguments > > came from where: some arguments might be inputs and some might be > extra > > configuration. We have currently decided that this distinguishes the > > two is > > cumbersome and not useful. If it does become important, we will need > to > > revisit this logic. > > > > > > On Tue, May 23, 2017 at 11:09 AM, Tal Liron <[email protected]> wrote: > > > > > Why do you think that on-the-fly updates won't require implementation > and > > > inputs? It's true that the execution plugin squashes inputs into > > arguments > > > and keeps the implementation in arguments.process.script_path, but > that's > > > an implementation detail. Some plugins may care more about what is an > > input > > > and what isn't, or do other kinds of mangling to the implementation. In > > any > > > case, the user should have to know how to massage these values: they > can > > > set up inputs and possibly configuration (same way as in TOSCA) and > then > > > call Operation.configure to do the right thing. > > > > > > The implementation and function are not the same, as you can see even > in > > > my trivial example. The implementation is the path to a script. The > > > function is a path to a Python function. "function" is absolutely not > an > > > evaluation of "implementation": for the execution plugin the function > is > > > either run_script_local or run_script_with_ssh, which is entirely > > > determined by the topology (relationship to a host node) and has > nothing > > to > > > do with the implementation. Who knows, some plugins in some cases might > > > totally throw away the implementation string in some cases if it > becomes > > > irrelevant in some configurations. > > > > > > I just don't see why we need to introduce restrictions only to avoid > > > saving a single text column here. As for separating inputs -- seems > very > > > obvious to me. > > > > > > We will follow up the discussion in person and post our final decision > > > here. > > > > > > On Tue, May 23, 2017 at 8:57 AM, Ran Ziv <[email protected]> wrote: > > > > > >> Actually I didn't mean ARIA-180 has already been merged, just that it > > >> might > > >> conflict with it and I think you two should sync about this. > > >> > > >> Regarding the new fields - I completely agree with your first > paragraph > > - > > >> the original values should indeed be kept, and generally speaking the > > >> service instance models should be decoupled from the template models > in > > >> order to allow on-the-fly updates and so on - but does this really > apply > > >> here? What is the value of storing the original user values on the > > service > > >> instance models in this case? On-the-fly updates won't require the > > >> original > > >> value, and it does make sense to me that the only place the original > > >> values > > >> would be stored is on the template. At the end of the day the > > >> `implementation` and the `function` are one and the same - its simply > > that > > >> one of them is already evaluated and the other isn't. Also, Is it not > > the > > >> same case as properties that use the `get_input` intrinsic function? > > >> > > >> I guess the upside just isn't that clear to me here, and there is the > > >> downside of these being pretty confusing. The way I see it it's as if > > >> "implementation" is called "user_implementation" (vs "actual > > >> implementation") , or rather "template_implementation" - which again > > hints > > >> it should sit over at the template models..? > > >> > > >> > > >> > > >> > > >> > > >> On Mon, May 22, 2017 at 7:07 PM, Tal Liron <[email protected]> > wrote: > > >> > > >> > I see Avia merged ARIA-180, so definitely will convert to > one-to-many > > >> now. > > >> > > > >> > The reason the original values are preserved are the same reason all > > our > > >> > original values are preserved. The idea is that we don't want to > make > > >> > service instance models dependent on service template models. They > are > > >> > instantiated from them, but these foreign-key fields are all > nullable. > > >> The > > >> > reason is to allow freedom for orchestration policies with > on-the-fly > > >> > changes the topology. For example, a special kind of scaling policy > > >> might > > >> > involve adding an Nginx load balancer node for HTTP connections. > > >> However, > > >> > what if the user did not have a node template for that kind of the > > >> node? If > > >> > we require the creation of an OperationTemplate for every Operation, > > we > > >> > make this awkward, because it changes the original template. So, > > anyway, > > >> > this has been our principle until now and I suggest we continue this > > for > > >> > operations, too. > > >> > > > >> > Let's walk through a simple example. > > >> > > > >> > 1. A service template with a simple compute node, with a single > > >> "dazzle" > > >> > operation on a custom interface, and this "dazzle" has a single > > >> > implementation string, "dazzle_node.sh", with a single string > > input: > > >> > "duration"="forever". To keep things simple, no special > > >> configuration is > > >> > added via dependencies. > > >> > 2. The parser will create an OperationTemplate with these values: > > >> > 1. implementation = "dazzle_node.sh" > > >> > 2. inputs = { > > >> > 1. duration = "forever"} > > >> > 3. plugin_specification = None > > >> > 4. configuration = {} > > >> > 3. During initial instantiation phase, the above values will > be > > >> > copied to Operation model as is. At this early stage, "function" > > and > > >> > "arguments" are still both empty. > > >> > 4. Now during instantiation *configuration* phase, we see that > this > > >> > operation uses the default execution plugin > > >> (plugin_specification=None). > > >> > Because this node is a host, it will be configured to be a > *remote* > > >> > operation with very specific arguments. Also, we decided that for > > all > > >> > plugins inputs get sent as *extra* arguments. So, here are all > > final > > >> > values for Operation: > > >> > 1. implementation = "dazzle_node.sh" > > >> > 2. inputs = { > > >> > 1. duration = "forever"} > > >> > 3. configuration = {} > > >> > 4. plugin = None > > >> > 5. function = "aria.orchestrator.execution_ > > >> > plugin.operations.run_script_with_ssh" > > >> > 6. arguments = { > > >> > 1. process = {} > > >> > 2. script_path = "create_node.sh" > > >> > 3. use_sudo = False > > >> > 4. hide_output = [] > > >> > 5. fabric_env = {user: DEFAULT_USER, password: > > >> DEFAULT_PASSWORD, > > >> > key: None, key_filename: None} > > >> > 6. duration = forever} > > >> > > > >> > So, you can see here that if we change "implementation" to be > > >> > "aria.orchestrator.execution_plugin.operations.run_script_with_ssh" > > >> (like > > >> > we did before), the implementation is hidden in one of the arguments > > >> > (script_path), which would be very tricky for users to discover. > > That's > > >> an > > >> > implementation detail of the execution plugin, not something you > want > > to > > >> > rely on. Also, if we merge everything into inputs, the original > > >> distinction > > >> > between what is an input and what is an implementation detail is > lost. > > >> > > > >> > The fields in this suggested scheme have distinct and clear > meanings: > > >> > > > >> > 1. "implementation" and "inputs" are directly from the TOSCA > spec. > > >> They > > >> > are preserved and never change. > > >> > 2. "configuration" is an extra optional ARIA extension dict > > allowing > > >> > users to configure how the plugin will execute the operation. > > >> > 3. "function" and "arguments" are internal implementation details > > of > > >> the > > >> > ARIA orchestrator. In a nicely straightforward sense they become > a > > >> > Python > > >> > function with kwargs. > > >> > > > >> > How this clears things up! > > >> > > > >> > On Sun, May 21, 2017 at 7:18 AM, Ran Ziv <[email protected]> > wrote: > > >> > > > >> > > +1 for examples, or perhaps a link to specific code sections. > > >> > > > > >> > > From what I did understand though, before we talk about additional > > >> > changes, > > >> > > I'm not at all sure I'm in favor of the current ones. > > >> > > > > >> > > First, the many-to-many relationship sounds in contrast to what > Avia > > >> is > > >> > > currently working on with (changing Parameters relationship type) > > >> > > > > >> > > Second, so what you're saying is that now Operation models will > have > > >> both > > >> > > "implementation" and "function", and both "inputs" and > "arguments"? > > >> This > > >> > > seems very confusing. > > >> > > I'm both not convinced the original values are needed at that > stage > > >> (can > > >> > we > > >> > > have an actual use case?), but even if they are, do we really need > > to > > >> > keep > > >> > > all four on the same model? > > >> > > How about having the original values on the OperationTemplate, and > > the > > >> > > "coerced values" (for lack of a better term..) on the Operation > > model? > > >> > > > > >> > > > > >> > > > > >> > > On Sun, May 21, 2017 at 11:45 AM, Arthur Berezin < > > >> [email protected]> > > >> > > wrote: > > >> > > > > >> > > > On Sat, May 20, 2017 at 3:47 AM Tal Liron <[email protected]> > > >> wrote: > > >> > > > > > >> > > > > Phew! Everything seems to work now: functions are parsed and > > >> > evaluated > > >> > > > > properly. But there are some changes that require explaining. > > >> > > > > > > >> > > > > The "configuration" field in OperationTemplate and Operation > > >> models > > >> > is > > >> > > > now > > >> > > > > a many-to-many with Parameter. This allows the values to fully > > >> > support > > >> > > > > intrinsic functions and types. > > >> > > > > > > >> > > > > And I made another change in modeling: Operation now also has > > >> > > "function" > > >> > > > > and "arguments" fields, which are what actually gets used by > the > > >> Task > > >> > > API > > >> > > > > instead of "implementation" and "inputs". This means that the > > >> > > > > "implementation" and "inputs" values are preserved as is. > During > > >> the > > >> > > > > configuration phase the "arguments" are created according to > the > > >> > usual > > >> > > > > logic: for the execution plugin, the "configuration" is > massaged > > >> > into a > > >> > > > few > > >> > > > > new "arguments", and otherwise other "inputs" are also merged > > into > > >> > > > > "arguments". > > >> > > > > > > >> > > > > Why make this change? Because I really think we should not > > mangle > > >> the > > >> > > > > "inputs": they are logically different from the arguments that > > get > > >> > sent > > >> > > > to > > >> > > > > the @operation function, even if in many cases we might treat > > them > > >> > > > > identically. Also, we cannot lose the "implementation" string: > > >> > plugins > > >> > > > will > > >> > > > > need to know what users put in there. The change is not big in > > >> terms > > >> > of > > >> > > > > extra code, but I think it helps make the code easier to > > >> understand: > > >> > > you > > >> > > > > can see exactly what is being merged into the final > "arguments" > > >> for > > >> > the > > >> > > > > task. The configure phase is now constructive rather than > > >> > destructive: > > >> > > > > existing fields are not changed. > > >> > > > > > > >> > > > > > >> > > > > > >> > > > Tal, could you provide several example models to showcase these > > >> > changes? > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > >> > > > > Before moving forward with a PR I want to see what other > > >> committers > > >> > > think > > >> > > > > about also renaming "implementation" and "inputs" in the Task > > API > > >> to > > >> > > > > "function" and "arguments". For now I'm leaving them as is, > but > > I > > >> > think > > >> > > > it > > >> > > > > would be even cleaner if they followed the new naming > convention > > >> in > > >> > > > > modeling. I continue being unhappy about how the Task API > treats > > >> both > > >> > > > > arguments and inputs as the same thing and squashing them all > > >> > > together... > > >> > > > > :( However, for now I would be happy if we just called them > > >> > > "arguments", > > >> > > > > because that's what they really are for us. > > >> > > > > > > >> > > > > -- > > >> > > > > Tal Liron > > >> > > > > Solution Architect > > >> > > > > [email protected] | +13123758299 <(312)%20375-8299> > > >> > > > > Cloudify | http://getcloudify.org > > >> > > > > < > > >> > > > > http://getcloudify.org?utm_source=signaturesatori&utm_ > > >> > > > medium=email&utm_campaign=general_signature > > >> > > > > > > > >> > > > > > > >> > > > > <https://twitter.com/CloudifySource> > > >> > > > > <https://www.linkedin.com/groups/8467478> > > >> > > > > <https://github.com/cloudify-cosmo> < > > >> https://github.com/cloudify- > > >> > > cosmo > > >> > > > > > > >> > > > > [image: Azure Webinar] > > >> > > > > < > > >> > > > > http://getcloudify.org/webinars/Azure-plugin-for- > > >> > > > cloudify-webinar.html?utm_source=signaturesatori&utm_ > > >> > > > medium=email&utm_campaign=general_signature > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Tal Liron > > >> > Senior Engineer > > >> > [email protected] | +1 (773) 828-9339 > > >> > Cloudify | http://getcloudify.org > > >> > <http://getcloudify.org?utm_source=signaturesatori&utm_ > > >> > medium=email&utm_campaign=general_signature> > > >> > > > >> > <https://twitter.com/CloudifySource> > > >> > <https://www.linkedin.com/groups/8467478> > > >> > <https://github.com/cloudify-cosmo> <https://github.com/cloudify- > > >> cosmo> > > >> > [image: Azure Webinar] > > >> > <http://getcloudify.org/webinars/Azure-plugin-for- > > >> > cloudify-webinar.html?utm_source=signaturesatori&utm_ > > >> > medium=email&utm_campaign=general_signature> > > >> > > > >> > > > > > > > > > > > > -- > > > Tal Liron > > > Senior Engineer > > > [email protected] | +1 (773) 828-9339 > > > Cloudify | http://getcloudify.org > > > <http://getcloudify.org?utm_source=signaturesatori&utm_ > > medium=email&utm_campaign=general_signature> > > > > > > <https://twitter.com/CloudifySource> > > > <https://www.linkedin.com/groups/8467478> > > > <https://github.com/cloudify-cosmo> <https://github.com/cloudify- > cosmo > > > > > > [image: Azure Webinar] > > > <http://getcloudify.org/webinars/Azure-plugin-for- > > cloudify-webinar.html?utm_source=signaturesatori&utm_ > > medium=email&utm_campaign=general_signature> > > > > > > > > > > > > > > -- > > Tal Liron > > Senior Engineer > > [email protected] | +1 (773) 828-9339 > > Cloudify | http://getcloudify.org > > <http://getcloudify.org?utm_source=signaturesatori&utm_ > > medium=email&utm_campaign=general_signature> > > > > <https://twitter.com/CloudifySource> > > <https://www.linkedin.com/groups/8467478> > > <https://github.com/cloudify-cosmo> <https://github.com/cloudify-cosmo > > > > [image: Azure Webinar] > > <http://getcloudify.org/webinars/Azure-plugin-for- > > cloudify-webinar.html?utm_source=signaturesatori&utm_ > > medium=email&utm_campaign=general_signature> > > > -- Tal Liron Senior Engineer [email protected] | +1 (773) 828-9339 Cloudify | http://getcloudify.org <http://getcloudify.org?utm_source=signaturesatori&utm_medium=email&utm_campaign=general_signature> <https://twitter.com/CloudifySource> <https://www.linkedin.com/groups/8467478> <https://github.com/cloudify-cosmo> <https://github.com/cloudify-cosmo> [image: Azure Webinar] <http://getcloudify.org/webinars/Azure-plugin-for-cloudify-webinar.html?utm_source=signaturesatori&utm_medium=email&utm_campaign=general_signature>
