Not entirely sure why the "implementation" gets its own field. It basically another input to the operation. Though it's a special input (since it enables for the plugin do decide which "function" to run), it's still an inputs. feel we already have a lot of different possible inputs to an operation: (1) interface inputs, (2) operation inputs (3) task inputs. I feel that the "implementation" should have a special place under the operation inputs, but not a field on its own...
On Thu, May 25, 2017 at 1:22 AM, Tal Liron <[email protected]> wrote: > We had a long face-to-face today and reached an alignment. ARIA-149 is > being updated to reflect it. > > So, we are going all the way with this: > > The "implementation" and "inputs" fields in the models refer *only* to what > comes from the TOSCA parser. They are passed from template to instance as > is. This means that "implementation" should always be considered an > artifact name. (Indeed, when we implement artifact in the near future, this > field may be refactored in some way, possibly to become a foreign key to an > artifact-related model.) > > The OperationTemplate and Operation models both also have a "configuration" > field, a dict of special parameters used as hints for orchestration. The > parser can generate them using our magical "dependencies" grammar > extensions. They are normal parameters (typed, and can contain intrinsic > functions). > > And, we are introducing two *new* fields to Operation and Task models (and > the related Task API): "function" and "arguments". "function" is the full > path to a Python function, and "arguments" is a dict of parameters that are > sent as arguments to the function. These are ARIA's implementation details > for orchestration, and are *not* directly related to TOSCA's > "implementation" and "inputs" fields, and indeed the Task model does not > have nor need the "implementation" and "inputs" (to put it another way: > we've renamed these two fields in Task, but they have the exact same usage > as before). > > The logic for tasks has not changed, but let's go over it again for > clarification with the new fields in mind: > > 1. The parser generates an OperationTemplate model, and sets the > "implementation" and "inputs" fields. Two ARIA grammar extensions > optionally apply: > 1. If the ">" is used in the "implementation", then the parser actually > leaves "implementation" empty (there is no artifact) and instead sets > "function". > 2. If the ">" is used in "dependencies", they are converted to > "configuration". > 2. During instantiation, all these fields are copied as is from > OperationTemplate to Operation. > 3. During the configuration phase of instantiation, there are two > possible routes: > 1. For the default execution plugin, it generates "function" and > "arguments" according to whether the operation is local or remote, > using > params from "configuration" if they are available (and valid). Also, > any > extra "configuration" params set by the user, which are not > recognized by > the plugin, just get appended to "arguments". > 2. For other plugins, we already have "function" set for us by the > parser (according to what the user put in "implementation" after > the ">"). > So, we just append all "configuration" params if they exist to > "arguments" > 4. For all plugins, "inputs" are also appended to "arguments". > > Some implications of this logic worth considering: > > 1. "implementation" and "inputs" are never mangled and always adhere to > the strict TOSCA typed interface contract. > 2. It's possible to use our special "dependencies" grammar to send > *extra > arbitrary* arguments to the Python operation function, *beyond* what is > allowed by the strict TOSCA contract. > 3. Python operation functions do not have a way to know which arguments > came from where: some arguments might be inputs and some might be extra > configuration. We have currently decided that this distinguishes the > two is > cumbersome and not useful. If it does become important, we will need to > revisit this logic. > > > On Tue, May 23, 2017 at 11:09 AM, Tal Liron <[email protected]> wrote: > > > Why do you think that on-the-fly updates won't require implementation and > > inputs? It's true that the execution plugin squashes inputs into > arguments > > and keeps the implementation in arguments.process.script_path, but that's > > an implementation detail. Some plugins may care more about what is an > input > > and what isn't, or do other kinds of mangling to the implementation. In > any > > case, the user should have to know how to massage these values: they can > > set up inputs and possibly configuration (same way as in TOSCA) and then > > call Operation.configure to do the right thing. > > > > The implementation and function are not the same, as you can see even in > > my trivial example. The implementation is the path to a script. The > > function is a path to a Python function. "function" is absolutely not an > > evaluation of "implementation": for the execution plugin the function is > > either run_script_local or run_script_with_ssh, which is entirely > > determined by the topology (relationship to a host node) and has nothing > to > > do with the implementation. Who knows, some plugins in some cases might > > totally throw away the implementation string in some cases if it becomes > > irrelevant in some configurations. > > > > I just don't see why we need to introduce restrictions only to avoid > > saving a single text column here. As for separating inputs -- seems very > > obvious to me. > > > > We will follow up the discussion in person and post our final decision > > here. > > > > On Tue, May 23, 2017 at 8:57 AM, Ran Ziv <[email protected]> wrote: > > > >> Actually I didn't mean ARIA-180 has already been merged, just that it > >> might > >> conflict with it and I think you two should sync about this. > >> > >> Regarding the new fields - I completely agree with your first paragraph > - > >> the original values should indeed be kept, and generally speaking the > >> service instance models should be decoupled from the template models in > >> order to allow on-the-fly updates and so on - but does this really apply > >> here? What is the value of storing the original user values on the > service > >> instance models in this case? On-the-fly updates won't require the > >> original > >> value, and it does make sense to me that the only place the original > >> values > >> would be stored is on the template. At the end of the day the > >> `implementation` and the `function` are one and the same - its simply > that > >> one of them is already evaluated and the other isn't. Also, Is it not > the > >> same case as properties that use the `get_input` intrinsic function? > >> > >> I guess the upside just isn't that clear to me here, and there is the > >> downside of these being pretty confusing. The way I see it it's as if > >> "implementation" is called "user_implementation" (vs "actual > >> implementation") , or rather "template_implementation" - which again > hints > >> it should sit over at the template models..? > >> > >> > >> > >> > >> > >> On Mon, May 22, 2017 at 7:07 PM, Tal Liron <[email protected]> wrote: > >> > >> > I see Avia merged ARIA-180, so definitely will convert to one-to-many > >> now. > >> > > >> > The reason the original values are preserved are the same reason all > our > >> > original values are preserved. The idea is that we don't want to make > >> > service instance models dependent on service template models. They are > >> > instantiated from them, but these foreign-key fields are all nullable. > >> The > >> > reason is to allow freedom for orchestration policies with on-the-fly > >> > changes the topology. For example, a special kind of scaling policy > >> might > >> > involve adding an Nginx load balancer node for HTTP connections. > >> However, > >> > what if the user did not have a node template for that kind of the > >> node? If > >> > we require the creation of an OperationTemplate for every Operation, > we > >> > make this awkward, because it changes the original template. So, > anyway, > >> > this has been our principle until now and I suggest we continue this > for > >> > operations, too. > >> > > >> > Let's walk through a simple example. > >> > > >> > 1. A service template with a simple compute node, with a single > >> "dazzle" > >> > operation on a custom interface, and this "dazzle" has a single > >> > implementation string, "dazzle_node.sh", with a single string > input: > >> > "duration"="forever". To keep things simple, no special > >> configuration is > >> > added via dependencies. > >> > 2. The parser will create an OperationTemplate with these values: > >> > 1. implementation = "dazzle_node.sh" > >> > 2. inputs = { > >> > 1. duration = "forever"} > >> > 3. plugin_specification = None > >> > 4. configuration = {} > >> > 3. During initial instantiation phase, the above values will be > >> > copied to Operation model as is. At this early stage, "function" > and > >> > "arguments" are still both empty. > >> > 4. Now during instantiation *configuration* phase, we see that this > >> > operation uses the default execution plugin > >> (plugin_specification=None). > >> > Because this node is a host, it will be configured to be a *remote* > >> > operation with very specific arguments. Also, we decided that for > all > >> > plugins inputs get sent as *extra* arguments. So, here are all > final > >> > values for Operation: > >> > 1. implementation = "dazzle_node.sh" > >> > 2. inputs = { > >> > 1. duration = "forever"} > >> > 3. configuration = {} > >> > 4. plugin = None > >> > 5. function = "aria.orchestrator.execution_ > >> > plugin.operations.run_script_with_ssh" > >> > 6. arguments = { > >> > 1. process = {} > >> > 2. script_path = "create_node.sh" > >> > 3. use_sudo = False > >> > 4. hide_output = [] > >> > 5. fabric_env = {user: DEFAULT_USER, password: > >> DEFAULT_PASSWORD, > >> > key: None, key_filename: None} > >> > 6. duration = forever} > >> > > >> > So, you can see here that if we change "implementation" to be > >> > "aria.orchestrator.execution_plugin.operations.run_script_with_ssh" > >> (like > >> > we did before), the implementation is hidden in one of the arguments > >> > (script_path), which would be very tricky for users to discover. > That's > >> an > >> > implementation detail of the execution plugin, not something you want > to > >> > rely on. Also, if we merge everything into inputs, the original > >> distinction > >> > between what is an input and what is an implementation detail is lost. > >> > > >> > The fields in this suggested scheme have distinct and clear meanings: > >> > > >> > 1. "implementation" and "inputs" are directly from the TOSCA spec. > >> They > >> > are preserved and never change. > >> > 2. "configuration" is an extra optional ARIA extension dict > allowing > >> > users to configure how the plugin will execute the operation. > >> > 3. "function" and "arguments" are internal implementation details > of > >> the > >> > ARIA orchestrator. In a nicely straightforward sense they become a > >> > Python > >> > function with kwargs. > >> > > >> > How this clears things up! > >> > > >> > On Sun, May 21, 2017 at 7:18 AM, Ran Ziv <[email protected]> wrote: > >> > > >> > > +1 for examples, or perhaps a link to specific code sections. > >> > > > >> > > From what I did understand though, before we talk about additional > >> > changes, > >> > > I'm not at all sure I'm in favor of the current ones. > >> > > > >> > > First, the many-to-many relationship sounds in contrast to what Avia > >> is > >> > > currently working on with (changing Parameters relationship type) > >> > > > >> > > Second, so what you're saying is that now Operation models will have > >> both > >> > > "implementation" and "function", and both "inputs" and "arguments"? > >> This > >> > > seems very confusing. > >> > > I'm both not convinced the original values are needed at that stage > >> (can > >> > we > >> > > have an actual use case?), but even if they are, do we really need > to > >> > keep > >> > > all four on the same model? > >> > > How about having the original values on the OperationTemplate, and > the > >> > > "coerced values" (for lack of a better term..) on the Operation > model? > >> > > > >> > > > >> > > > >> > > On Sun, May 21, 2017 at 11:45 AM, Arthur Berezin < > >> [email protected]> > >> > > wrote: > >> > > > >> > > > On Sat, May 20, 2017 at 3:47 AM Tal Liron <[email protected]> > >> wrote: > >> > > > > >> > > > > Phew! Everything seems to work now: functions are parsed and > >> > evaluated > >> > > > > properly. But there are some changes that require explaining. > >> > > > > > >> > > > > The "configuration" field in OperationTemplate and Operation > >> models > >> > is > >> > > > now > >> > > > > a many-to-many with Parameter. This allows the values to fully > >> > support > >> > > > > intrinsic functions and types. > >> > > > > > >> > > > > And I made another change in modeling: Operation now also has > >> > > "function" > >> > > > > and "arguments" fields, which are what actually gets used by the > >> Task > >> > > API > >> > > > > instead of "implementation" and "inputs". This means that the > >> > > > > "implementation" and "inputs" values are preserved as is. During > >> the > >> > > > > configuration phase the "arguments" are created according to the > >> > usual > >> > > > > logic: for the execution plugin, the "configuration" is massaged > >> > into a > >> > > > few > >> > > > > new "arguments", and otherwise other "inputs" are also merged > into > >> > > > > "arguments". > >> > > > > > >> > > > > Why make this change? Because I really think we should not > mangle > >> the > >> > > > > "inputs": they are logically different from the arguments that > get > >> > sent > >> > > > to > >> > > > > the @operation function, even if in many cases we might treat > them > >> > > > > identically. Also, we cannot lose the "implementation" string: > >> > plugins > >> > > > will > >> > > > > need to know what users put in there. The change is not big in > >> terms > >> > of > >> > > > > extra code, but I think it helps make the code easier to > >> understand: > >> > > you > >> > > > > can see exactly what is being merged into the final "arguments" > >> for > >> > the > >> > > > > task. The configure phase is now constructive rather than > >> > destructive: > >> > > > > existing fields are not changed. > >> > > > > > >> > > > > >> > > > > >> > > > Tal, could you provide several example models to showcase these > >> > changes? > >> > > > > >> > > > > >> > > > > >> > > > > > >> > > > > Before moving forward with a PR I want to see what other > >> committers > >> > > think > >> > > > > about also renaming "implementation" and "inputs" in the Task > API > >> to > >> > > > > "function" and "arguments". For now I'm leaving them as is, but > I > >> > think > >> > > > it > >> > > > > would be even cleaner if they followed the new naming convention > >> in > >> > > > > modeling. I continue being unhappy about how the Task API treats > >> both > >> > > > > arguments and inputs as the same thing and squashing them all > >> > > together... > >> > > > > :( However, for now I would be happy if we just called them > >> > > "arguments", > >> > > > > because that's what they really are for us. > >> > > > > > >> > > > > -- > >> > > > > Tal Liron > >> > > > > Solution Architect > >> > > > > [email protected] | +13123758299 <(312)%20375-8299> > >> > > > > Cloudify | http://getcloudify.org > >> > > > > < > >> > > > > http://getcloudify.org?utm_source=signaturesatori&utm_ > >> > > > medium=email&utm_campaign=general_signature > >> > > > > > > >> > > > > > >> > > > > <https://twitter.com/CloudifySource> > >> > > > > <https://www.linkedin.com/groups/8467478> > >> > > > > <https://github.com/cloudify-cosmo> < > >> https://github.com/cloudify- > >> > > cosmo > >> > > > > > >> > > > > [image: Azure Webinar] > >> > > > > < > >> > > > > http://getcloudify.org/webinars/Azure-plugin-for- > >> > > > cloudify-webinar.html?utm_source=signaturesatori&utm_ > >> > > > medium=email&utm_campaign=general_signature > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Tal Liron > >> > Senior Engineer > >> > [email protected] | +1 (773) 828-9339 > >> > Cloudify | http://getcloudify.org > >> > <http://getcloudify.org?utm_source=signaturesatori&utm_ > >> > medium=email&utm_campaign=general_signature> > >> > > >> > <https://twitter.com/CloudifySource> > >> > <https://www.linkedin.com/groups/8467478> > >> > <https://github.com/cloudify-cosmo> <https://github.com/cloudify- > >> cosmo> > >> > [image: Azure Webinar] > >> > <http://getcloudify.org/webinars/Azure-plugin-for- > >> > cloudify-webinar.html?utm_source=signaturesatori&utm_ > >> > medium=email&utm_campaign=general_signature> > >> > > >> > > > > > > > > -- > > Tal Liron > > Senior Engineer > > [email protected] | +1 (773) 828-9339 > > Cloudify | http://getcloudify.org > > <http://getcloudify.org?utm_source=signaturesatori&utm_ > medium=email&utm_campaign=general_signature> > > > > <https://twitter.com/CloudifySource> > > <https://www.linkedin.com/groups/8467478> > > <https://github.com/cloudify-cosmo> <https://github.com/cloudify-cosmo > > > > [image: Azure Webinar] > > <http://getcloudify.org/webinars/Azure-plugin-for- > cloudify-webinar.html?utm_source=signaturesatori&utm_ > medium=email&utm_campaign=general_signature> > > > > > > > > -- > Tal Liron > Senior Engineer > [email protected] | +1 (773) 828-9339 > Cloudify | http://getcloudify.org > <http://getcloudify.org?utm_source=signaturesatori&utm_ > medium=email&utm_campaign=general_signature> > > <https://twitter.com/CloudifySource> > <https://www.linkedin.com/groups/8467478> > <https://github.com/cloudify-cosmo> <https://github.com/cloudify-cosmo> > [image: Azure Webinar] > <http://getcloudify.org/webinars/Azure-plugin-for- > cloudify-webinar.html?utm_source=signaturesatori&utm_ > medium=email&utm_campaign=general_signature> >
