We had a long face-to-face today and reached an alignment. ARIA-149 is
being updated to reflect it.
So, we are going all the way with this:
The "implementation" and "inputs" fields in the models refer *only* to what
comes from the TOSCA parser. They are passed from template to instance as
is. This means that "implementation" should always be considered an
artifact name. (Indeed, when we implement artifact in the near future, this
field may be refactored in some way, possibly to become a foreign key to an
artifact-related model.)
The OperationTemplate and Operation models both also have a "configuration"
field, a dict of special parameters used as hints for orchestration. The
parser can generate them using our magical "dependencies" grammar
extensions. They are normal parameters (typed, and can contain intrinsic
functions).
And, we are introducing two *new* fields to Operation and Task models (and
the related Task API): "function" and "arguments". "function" is the full
path to a Python function, and "arguments" is a dict of parameters that are
sent as arguments to the function. These are ARIA's implementation details
for orchestration, and are *not* directly related to TOSCA's
"implementation" and "inputs" fields, and indeed the Task model does not
have nor need the "implementation" and "inputs" (to put it another way:
we've renamed these two fields in Task, but they have the exact same usage
as before).
The logic for tasks has not changed, but let's go over it again for
clarification with the new fields in mind:
1. The parser generates an OperationTemplate model, and sets the
"implementation" and "inputs" fields. Two ARIA grammar extensions
optionally apply:
1. If the ">" is used in the "implementation", then the parser actually
leaves "implementation" empty (there is no artifact) and instead sets
"function".
2. If the ">" is used in "dependencies", they are converted to
"configuration".
2. During instantiation, all these fields are copied as is from
OperationTemplate to Operation.
3. During the configuration phase of instantiation, there are two
possible routes:
1. For the default execution plugin, it generates "function" and
"arguments" according to whether the operation is local or remote, using
params from "configuration" if they are available (and valid). Also, any
extra "configuration" params set by the user, which are not recognized by
the plugin, just get appended to "arguments".
2. For other plugins, we already have "function" set for us by the
parser (according to what the user put in "implementation" after
the ">").
So, we just append all "configuration" params if they exist to "arguments"
4. For all plugins, "inputs" are also appended to "arguments".
Some implications of this logic worth considering:
1. "implementation" and "inputs" are never mangled and always adhere to
the strict TOSCA typed interface contract.
2. It's possible to use our special "dependencies" grammar to send *extra
arbitrary* arguments to the Python operation function, *beyond* what is
allowed by the strict TOSCA contract.
3. Python operation functions do not have a way to know which arguments
came from where: some arguments might be inputs and some might be extra
configuration. We have currently decided that this distinguishes the two is
cumbersome and not useful. If it does become important, we will need to
revisit this logic.
On Tue, May 23, 2017 at 11:09 AM, Tal Liron <[email protected]> wrote:
> Why do you think that on-the-fly updates won't require implementation and
> inputs? It's true that the execution plugin squashes inputs into arguments
> and keeps the implementation in arguments.process.script_path, but that's
> an implementation detail. Some plugins may care more about what is an input
> and what isn't, or do other kinds of mangling to the implementation. In any
> case, the user should have to know how to massage these values: they can
> set up inputs and possibly configuration (same way as in TOSCA) and then
> call Operation.configure to do the right thing.
>
> The implementation and function are not the same, as you can see even in
> my trivial example. The implementation is the path to a script. The
> function is a path to a Python function. "function" is absolutely not an
> evaluation of "implementation": for the execution plugin the function is
> either run_script_local or run_script_with_ssh, which is entirely
> determined by the topology (relationship to a host node) and has nothing to
> do with the implementation. Who knows, some plugins in some cases might
> totally throw away the implementation string in some cases if it becomes
> irrelevant in some configurations.
>
> I just don't see why we need to introduce restrictions only to avoid
> saving a single text column here. As for separating inputs -- seems very
> obvious to me.
>
> We will follow up the discussion in person and post our final decision
> here.
>
> On Tue, May 23, 2017 at 8:57 AM, Ran Ziv <[email protected]> wrote:
>
>> Actually I didn't mean ARIA-180 has already been merged, just that it
>> might
>> conflict with it and I think you two should sync about this.
>>
>> Regarding the new fields - I completely agree with your first paragraph -
>> the original values should indeed be kept, and generally speaking the
>> service instance models should be decoupled from the template models in
>> order to allow on-the-fly updates and so on - but does this really apply
>> here? What is the value of storing the original user values on the service
>> instance models in this case? On-the-fly updates won't require the
>> original
>> value, and it does make sense to me that the only place the original
>> values
>> would be stored is on the template. At the end of the day the
>> `implementation` and the `function` are one and the same - its simply that
>> one of them is already evaluated and the other isn't. Also, Is it not the
>> same case as properties that use the `get_input` intrinsic function?
>>
>> I guess the upside just isn't that clear to me here, and there is the
>> downside of these being pretty confusing. The way I see it it's as if
>> "implementation" is called "user_implementation" (vs "actual
>> implementation") , or rather "template_implementation" - which again hints
>> it should sit over at the template models..?
>>
>>
>>
>>
>>
>> On Mon, May 22, 2017 at 7:07 PM, Tal Liron <[email protected]> wrote:
>>
>> > I see Avia merged ARIA-180, so definitely will convert to one-to-many
>> now.
>> >
>> > The reason the original values are preserved are the same reason all our
>> > original values are preserved. The idea is that we don't want to make
>> > service instance models dependent on service template models. They are
>> > instantiated from them, but these foreign-key fields are all nullable.
>> The
>> > reason is to allow freedom for orchestration policies with on-the-fly
>> > changes the topology. For example, a special kind of scaling policy
>> might
>> > involve adding an Nginx load balancer node for HTTP connections.
>> However,
>> > what if the user did not have a node template for that kind of the
>> node? If
>> > we require the creation of an OperationTemplate for every Operation, we
>> > make this awkward, because it changes the original template. So, anyway,
>> > this has been our principle until now and I suggest we continue this for
>> > operations, too.
>> >
>> > Let's walk through a simple example.
>> >
>> > 1. A service template with a simple compute node, with a single
>> "dazzle"
>> > operation on a custom interface, and this "dazzle" has a single
>> > implementation string, "dazzle_node.sh", with a single string input:
>> > "duration"="forever". To keep things simple, no special
>> configuration is
>> > added via dependencies.
>> > 2. The parser will create an OperationTemplate with these values:
>> > 1. implementation = "dazzle_node.sh"
>> > 2. inputs = {
>> > 1. duration = "forever"}
>> > 3. plugin_specification = None
>> > 4. configuration = {}
>> > 3. During initial instantiation phase, the above values will be
>> > copied to Operation model as is. At this early stage, "function" and
>> > "arguments" are still both empty.
>> > 4. Now during instantiation *configuration* phase, we see that this
>> > operation uses the default execution plugin
>> (plugin_specification=None).
>> > Because this node is a host, it will be configured to be a *remote*
>> > operation with very specific arguments. Also, we decided that for all
>> > plugins inputs get sent as *extra* arguments. So, here are all final
>> > values for Operation:
>> > 1. implementation = "dazzle_node.sh"
>> > 2. inputs = {
>> > 1. duration = "forever"}
>> > 3. configuration = {}
>> > 4. plugin = None
>> > 5. function = "aria.orchestrator.execution_
>> > plugin.operations.run_script_with_ssh"
>> > 6. arguments = {
>> > 1. process = {}
>> > 2. script_path = "create_node.sh"
>> > 3. use_sudo = False
>> > 4. hide_output = []
>> > 5. fabric_env = {user: DEFAULT_USER, password:
>> DEFAULT_PASSWORD,
>> > key: None, key_filename: None}
>> > 6. duration = forever}
>> >
>> > So, you can see here that if we change "implementation" to be
>> > "aria.orchestrator.execution_plugin.operations.run_script_with_ssh"
>> (like
>> > we did before), the implementation is hidden in one of the arguments
>> > (script_path), which would be very tricky for users to discover. That's
>> an
>> > implementation detail of the execution plugin, not something you want to
>> > rely on. Also, if we merge everything into inputs, the original
>> distinction
>> > between what is an input and what is an implementation detail is lost.
>> >
>> > The fields in this suggested scheme have distinct and clear meanings:
>> >
>> > 1. "implementation" and "inputs" are directly from the TOSCA spec.
>> They
>> > are preserved and never change.
>> > 2. "configuration" is an extra optional ARIA extension dict allowing
>> > users to configure how the plugin will execute the operation.
>> > 3. "function" and "arguments" are internal implementation details of
>> the
>> > ARIA orchestrator. In a nicely straightforward sense they become a
>> > Python
>> > function with kwargs.
>> >
>> > How this clears things up!
>> >
>> > On Sun, May 21, 2017 at 7:18 AM, Ran Ziv <[email protected]> wrote:
>> >
>> > > +1 for examples, or perhaps a link to specific code sections.
>> > >
>> > > From what I did understand though, before we talk about additional
>> > changes,
>> > > I'm not at all sure I'm in favor of the current ones.
>> > >
>> > > First, the many-to-many relationship sounds in contrast to what Avia
>> is
>> > > currently working on with (changing Parameters relationship type)
>> > >
>> > > Second, so what you're saying is that now Operation models will have
>> both
>> > > "implementation" and "function", and both "inputs" and "arguments"?
>> This
>> > > seems very confusing.
>> > > I'm both not convinced the original values are needed at that stage
>> (can
>> > we
>> > > have an actual use case?), but even if they are, do we really need to
>> > keep
>> > > all four on the same model?
>> > > How about having the original values on the OperationTemplate, and the
>> > > "coerced values" (for lack of a better term..) on the Operation model?
>> > >
>> > >
>> > >
>> > > On Sun, May 21, 2017 at 11:45 AM, Arthur Berezin <
>> [email protected]>
>> > > wrote:
>> > >
>> > > > On Sat, May 20, 2017 at 3:47 AM Tal Liron <[email protected]>
>> wrote:
>> > > >
>> > > > > Phew! Everything seems to work now: functions are parsed and
>> > evaluated
>> > > > > properly. But there are some changes that require explaining.
>> > > > >
>> > > > > The "configuration" field in OperationTemplate and Operation
>> models
>> > is
>> > > > now
>> > > > > a many-to-many with Parameter. This allows the values to fully
>> > support
>> > > > > intrinsic functions and types.
>> > > > >
>> > > > > And I made another change in modeling: Operation now also has
>> > > "function"
>> > > > > and "arguments" fields, which are what actually gets used by the
>> Task
>> > > API
>> > > > > instead of "implementation" and "inputs". This means that the
>> > > > > "implementation" and "inputs" values are preserved as is. During
>> the
>> > > > > configuration phase the "arguments" are created according to the
>> > usual
>> > > > > logic: for the execution plugin, the "configuration" is massaged
>> > into a
>> > > > few
>> > > > > new "arguments", and otherwise other "inputs" are also merged into
>> > > > > "arguments".
>> > > > >
>> > > > > Why make this change? Because I really think we should not mangle
>> the
>> > > > > "inputs": they are logically different from the arguments that get
>> > sent
>> > > > to
>> > > > > the @operation function, even if in many cases we might treat them
>> > > > > identically. Also, we cannot lose the "implementation" string:
>> > plugins
>> > > > will
>> > > > > need to know what users put in there. The change is not big in
>> terms
>> > of
>> > > > > extra code, but I think it helps make the code easier to
>> understand:
>> > > you
>> > > > > can see exactly what is being merged into the final "arguments"
>> for
>> > the
>> > > > > task. The configure phase is now constructive rather than
>> > destructive:
>> > > > > existing fields are not changed.
>> > > > >
>> > > >
>> > > >
>> > > > Tal, could you provide several example models to showcase these
>> > changes?
>> > > >
>> > > >
>> > > >
>> > > > >
>> > > > > Before moving forward with a PR I want to see what other
>> committers
>> > > think
>> > > > > about also renaming "implementation" and "inputs" in the Task API
>> to
>> > > > > "function" and "arguments". For now I'm leaving them as is, but I
>> > think
>> > > > it
>> > > > > would be even cleaner if they followed the new naming convention
>> in
>> > > > > modeling. I continue being unhappy about how the Task API treats
>> both
>> > > > > arguments and inputs as the same thing and squashing them all
>> > > together...
>> > > > > :( However, for now I would be happy if we just called them
>> > > "arguments",
>> > > > > because that's what they really are for us.
>> > > > >
>> > > > > --
>> > > > > Tal Liron
>> > > > > Solution Architect
>> > > > > [email protected] | +13123758299 <(312)%20375-8299>
>> > > > > Cloudify | http://getcloudify.org
>> > > > > <
>> > > > > http://getcloudify.org?utm_source=signaturesatori&utm_
>> > > > medium=email&utm_campaign=general_signature
>> > > > > >
>> > > > >
>> > > > > <https://twitter.com/CloudifySource>
>> > > > > <https://www.linkedin.com/groups/8467478>
>> > > > > <https://github.com/cloudify-cosmo> <
>> https://github.com/cloudify-
>> > > cosmo
>> > > > >
>> > > > > [image: Azure Webinar]
>> > > > > <
>> > > > > http://getcloudify.org/webinars/Azure-plugin-for-
>> > > > cloudify-webinar.html?utm_source=signaturesatori&utm_
>> > > > medium=email&utm_campaign=general_signature
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Tal Liron
>> > Senior Engineer
>> > [email protected] | +1 (773) 828-9339
>> > Cloudify | http://getcloudify.org
>> > <http://getcloudify.org?utm_source=signaturesatori&utm_
>> > medium=email&utm_campaign=general_signature>
>> >
>> > <https://twitter.com/CloudifySource>
>> > <https://www.linkedin.com/groups/8467478>
>> > <https://github.com/cloudify-cosmo> <https://github.com/cloudify-
>> cosmo>
>> > [image: Azure Webinar]
>> > <http://getcloudify.org/webinars/Azure-plugin-for-
>> > cloudify-webinar.html?utm_source=signaturesatori&utm_
>> > medium=email&utm_campaign=general_signature>
>> >
>>
>
>
>
> --
> Tal Liron
> Senior Engineer
> [email protected] | +1 (773) 828-9339
> Cloudify | http://getcloudify.org
> <http://getcloudify.org?utm_source=signaturesatori&utm_medium=email&utm_campaign=general_signature>
>
> <https://twitter.com/CloudifySource>
> <https://www.linkedin.com/groups/8467478>
> <https://github.com/cloudify-cosmo> <https://github.com/cloudify-cosmo>
> [image: Azure Webinar]
> <http://getcloudify.org/webinars/Azure-plugin-for-cloudify-webinar.html?utm_source=signaturesatori&utm_medium=email&utm_campaign=general_signature>
>
>
--
Tal Liron
Senior Engineer
[email protected] | +1 (773) 828-9339
Cloudify | http://getcloudify.org
<http://getcloudify.org?utm_source=signaturesatori&utm_medium=email&utm_campaign=general_signature>
<https://twitter.com/CloudifySource>
<https://www.linkedin.com/groups/8467478>
<https://github.com/cloudify-cosmo> <https://github.com/cloudify-cosmo>
[image: Azure Webinar]
<http://getcloudify.org/webinars/Azure-plugin-for-cloudify-webinar.html?utm_source=signaturesatori&utm_medium=email&utm_campaign=general_signature>
