Re: Workflow graph, Juju charm and node states

[Tal Liron]

>
> 1) Is it possible to send me an example of a custom workflow graph?
>

There is a rather simple one here:

https://github.com/apache/incubator-ariatosca/tree/master/tests/resources/service-templates/tosca-simple-1.0/node-cellar

On the YAML side, you'll see how we use the aria.Workflow policy type to
link the workflow function. In this example, we're deriving the type in
order to add an extra property, but you can also use the aria.Workflow type
directly if you don't need to extend it.

Then, in workflows.py you will see the actual Python function that builds
the task graph. It's a very trivial one in this case: we simply go through
all nodes and try to create a task to execute the operation. Nodes that
don't have the interface/operation will raise an exception, so we just
ignore those and move on.

The task graph API is much richer and allows complex interdependencies. At
this point I don't think we have a good example for more complex workflows.

2) When executing an operation associated with a "script", what are the
> main implications or differences between executing a Juju charm vs
> executing a shell script.
>
> Is it something like this?
>
> - a shell script is an artifact that is included in the CSAR and is likely
> executed by the local TOSCA ARIA orchestrator
>
> - a juju charm is not an artifact (therefore not included in the CSAR) and
> is likely executed by a remote Juju service orchestrator
>

It's not just a shell script -- it would be any OS executable. It could be
a shell script, or a Python script, or an .exe, etc.

You are right about Juju not being an artifact. This is why I am arguing
against the current idea on the table for TOSCA 1.2 to support more complex
operations using artifacts: a charm is just not an artifact. It should be,
in my opinion, solved by new operation types.

Until then, ARIA "solves" the situation by allowing for plugins that would
provide their own execution of the operation. We don't have a Juju plugin
right now, but I think it would not be hard at all to create one. It's
something you can contribute!


> 3) Is it possible to create custom node states using TOSCA/ARIA?
>

ARIA does not currently allow for this, because TOSCA doesn't. So we indeed
validate that the state you set is one of the currently supported states.
(Actually there is one extra state that ARIA adds internally, but in any
case a user can't simply add one.)

A workaround is to use a custom node attribute instead. You can then set
this "state" attribute to anything you wish. Operations can set this
attribute using ctx. So in a bash script it would like this:

ctx node attribute state = my-custom-state

More about using ctx:
https://cwiki.apache.org/confluence/display/ARIATOSCA/Execution+Context

The downside is that you would have to declare this attribute at the node
type level, so it won't work with built-in nodes unless you derive from
them.

4) Also related to node states. It looks like the YAML specs has a couple
> of discrepancies for the normative uninstall workflow (section 5.7.4.4.2 in
> YAML 1.0).
>
> IMO the available state is not defined and the diagram at the top of the
> workflow should use the started state instead.
>
> IMO the configured state at the bottom of the workflow diagram should be
> replaced with the initial state instead.
>
> Do you agree?
>

I agree. :) The spec actually only really discusses the install workflow,
not the others, and even with install it has two separate diagrams that
contradict each other.

Changing the node state is currently handled in ARIA via event listeners:
certain kinds of tasks trigger events that cause the state to change. The
mechanism is currently a bit opaque and not very easy for users to extend.
Perhaps other ARIA committers can comment more about it?

Workflow graph, Juju charm and node states

[Steve Baillargeon]

Hi Tal

1) Is it possible to send me an example of a custom workflow graph?

I am trying to visualize such graph or use it in my presentation



2) When executing an operation associated with a "script", what are the main 
implications or differences between executing a Juju charm vs executing a shell 
script.

Is it something like this?

- a shell script is an artifact that is included in the CSAR and is likely 
executed by the local TOSCA ARIA orchestrator

- a juju charm is not an artifact (therefore not included in the CSAR) and is 
likely executed by a remote Juju service orchestrator



3) Is it possible to create custom node states using TOSCA/ARIA?

I suspect custom node states cannot be defined in the TOSCA service template 
and must be implemented directly by the TOSCA orchestrator (?).



4) Also related to node states. It looks like the YAML specs has a couple of 
discrepancies for the normative uninstall workflow (section 5.7.4.4.2 in YAML 
1.0).

IMO the available state is not defined and the diagram at the top of the 
workflow should use the started state instead.

IMO the configured state at the bottom of the workflow diagram should be 
replaced with the initial state instead.

Do you agree?



Thanks

Regards
Steve B

Re: Service Composition / Substitution Mapping

[Tal Liron]

Our goal with adding new "conventions" to ARIA, such as policies, is to
always make them optional. The idea is that a plain-vanilla TOSCA template
would "just work" in ARIA via sensible defaults. The extra stuff is there
if you know you are using ARIA and you want to make use of its features.
(The opposite is true, too: we make sure that any additions are still pure
TOSCA and would be parsed validly by other TOSCA parsers.)

On Wed, Aug 2, 2017 at 9:08 AM, DeWayne Filppi  wrote:

> Cool.  Missed that.  That leaves things almost completely wide open from
> the orchestrator side, IOW few predefined keys.  Too few IMHO, but if
> everyone uses ARIA conventions it could work.
>
> On Tue, Aug 1, 2017 at 11:49 PM, Tal Liron  wrote:
>
> > I agree! Luckily metadata exists in the 1.0 spec. :)
> >
> > http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-
> > YAML/v1.0/cos01/TOSCA-Simple-Profile-YAML-v1.0-cos01.html#_Toc379455044
> >
> > On Tue, Aug 1, 2017 at 7:16 PM, DeWayne Filppi 
> > wrote:
> >
> > > It occurs that it might be useful to be able to tag service templates
> > with
> > > arbitrary meta-data.  Perhaps at one level carried forward from a CSAR
> > > manifest, but also user definable.  This would allow inter-service
> > > references to be definitive, if desired.  This could be implicitly
> > defined
> > > as a capability by the orchestrator, but some kind of special
> requirement
> > > type(s) would be needed to utilize it.  This way, external repos could
> be
> > > used safely and directly without the separate load step.
> > >
> > > On Tue, Aug 1, 2017 at 12:43 PM, Tal Liron  wrote:
> > >
> > > > Thanks for the kudos. :)
> > > >
> > > > This topic was discussed on this list a while ago. It's indeed tricky
> > to
> > > > get right, because TOSCA leaves a lot of room for the orchestrator to
> > > > implement.
> > > >
> > > > I'm thinking of it working something like this:
> > > >
> > > > 1. The reqs-and-caps engine by default will always look for
> satisfiable
> > > > capabilities within the currently instantiated service. HOWEVER, if
> > such
> > > a
> > > > capability is not present, the option is there to look for another
> > > > instantiated service that exposes the capabilities in substitution
> > > > mappings.
> > > >
> > > > 2. If we DON'T have another instantiated service, but DO have a
> service
> > > > template that could fit the bill, perhaps we need to instantiate that
> > > other
> > > > service first. One obvious option is to do this automatically. But I
> > feel
> > > > like this can create unforeseen consequences -- for example, some
> dummy
> > > > test template that someone happened to have in the database might get
> > > > instantiated by mistake. Also, it might need to trigger multiple
> > install
> > > > workflows at once... a big mess. So I suggest that instead we
> provide a
> > > > very detailed validation error here saying that the requirement
> cannot
> > be
> > > > satisfied, HOWEVER there exist service templates A, B, and C that can
> > > > substitute for us, so maybe the nice user would like to instantiate
> > them
> > > > first? This seems very reasonable to me.
> > > >
> > > > 3. If indeed another service satisfies this, a special node is added
> to
> > > the
> > > > current service (with the correct type -- but without a service
> > template
> > > > foreign key), which serves as a proxy of the other service template.
> > I'm
> > > > not sure how we would mark this exactly. We can't use the service_fk
> > > field,
> > > > because it's still in our current service. So perhaps there's need
> of a
> > > new
> > > > fk field, maybe substituted_service_fk?
> > > >
> > > > The above might be "sensible defaults," but it seems to me that users
> > > > really need control over this. So I propose to add a new
> > aria.Composition
> > > > policy that would let you provide hints for this mechanism. For
> > example,
> > > > you might want to "filter" the target service by service template
> name
> > > and
> > > > even by metadata in the service template. For example, you might want
> > to
> > > > require version 1.2.2 of a specific service, no less.
> > > >
> > > > Those are some quick thoughts. Exactly how such a policy would look
> > with
> > > > require more thought...
> > > >
> > > >
> > > > On Tue, Aug 1, 2017 at 2:20 PM, Avia Efrat  wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > I'm starting to work on a full implementation of
> > substitution_mapping,
> > > > > which will lead to the ability of service composition.
> > > > >
> > > > > For those unacquainted with substitution mapping, here are some
> quick
> > > > > resources:
> > > > > *From the spec
> > > > >  > > > > YAML/v1.0/TOSCA-Simple-Profile-YAML-v1.0.html>,
> > > > > sections:*
> > > > > 2.10
> > > > >  > 

Re: Service Composition / Substitution Mapping

[DeWayne Filppi]

Cool.  Missed that.  That leaves things almost completely wide open from
the orchestrator side, IOW few predefined keys.  Too few IMHO, but if
everyone uses ARIA conventions it could work.

On Tue, Aug 1, 2017 at 11:49 PM, Tal Liron  wrote:

> I agree! Luckily metadata exists in the 1.0 spec. :)
>
> http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-
> YAML/v1.0/cos01/TOSCA-Simple-Profile-YAML-v1.0-cos01.html#_Toc379455044
>
> On Tue, Aug 1, 2017 at 7:16 PM, DeWayne Filppi 
> wrote:
>
> > It occurs that it might be useful to be able to tag service templates
> with
> > arbitrary meta-data.  Perhaps at one level carried forward from a CSAR
> > manifest, but also user definable.  This would allow inter-service
> > references to be definitive, if desired.  This could be implicitly
> defined
> > as a capability by the orchestrator, but some kind of special requirement
> > type(s) would be needed to utilize it.  This way, external repos could be
> > used safely and directly without the separate load step.
> >
> > On Tue, Aug 1, 2017 at 12:43 PM, Tal Liron  wrote:
> >
> > > Thanks for the kudos. :)
> > >
> > > This topic was discussed on this list a while ago. It's indeed tricky
> to
> > > get right, because TOSCA leaves a lot of room for the orchestrator to
> > > implement.
> > >
> > > I'm thinking of it working something like this:
> > >
> > > 1. The reqs-and-caps engine by default will always look for satisfiable
> > > capabilities within the currently instantiated service. HOWEVER, if
> such
> > a
> > > capability is not present, the option is there to look for another
> > > instantiated service that exposes the capabilities in substitution
> > > mappings.
> > >
> > > 2. If we DON'T have another instantiated service, but DO have a service
> > > template that could fit the bill, perhaps we need to instantiate that
> > other
> > > service first. One obvious option is to do this automatically. But I
> feel
> > > like this can create unforeseen consequences -- for example, some dummy
> > > test template that someone happened to have in the database might get
> > > instantiated by mistake. Also, it might need to trigger multiple
> install
> > > workflows at once... a big mess. So I suggest that instead we provide a
> > > very detailed validation error here saying that the requirement cannot
> be
> > > satisfied, HOWEVER there exist service templates A, B, and C that can
> > > substitute for us, so maybe the nice user would like to instantiate
> them
> > > first? This seems very reasonable to me.
> > >
> > > 3. If indeed another service satisfies this, a special node is added to
> > the
> > > current service (with the correct type -- but without a service
> template
> > > foreign key), which serves as a proxy of the other service template.
> I'm
> > > not sure how we would mark this exactly. We can't use the service_fk
> > field,
> > > because it's still in our current service. So perhaps there's need of a
> > new
> > > fk field, maybe substituted_service_fk?
> > >
> > > The above might be "sensible defaults," but it seems to me that users
> > > really need control over this. So I propose to add a new
> aria.Composition
> > > policy that would let you provide hints for this mechanism. For
> example,
> > > you might want to "filter" the target service by service template name
> > and
> > > even by metadata in the service template. For example, you might want
> to
> > > require version 1.2.2 of a specific service, no less.
> > >
> > > Those are some quick thoughts. Exactly how such a policy would look
> with
> > > require more thought...
> > >
> > >
> > > On Tue, Aug 1, 2017 at 2:20 PM, Avia Efrat  wrote:
> > >
> > > > Hello all,
> > > >
> > > > I'm starting to work on a full implementation of
> substitution_mapping,
> > > > which will lead to the ability of service composition.
> > > >
> > > > For those unacquainted with substitution mapping, here are some quick
> > > > resources:
> > > > *From the spec
> > > >  > > > YAML/v1.0/TOSCA-Simple-Profile-YAML-v1.0.html>,
> > > > sections:*
> > > > 2.10
> > > >  > > > YAML/v1.0/os/TOSCA-Simple-Profile-YAML-v1.0-os.html#_Toc471725208>,
> > > > 2.11
> > > >  > > > YAML/v1.0/os/TOSCA-Simple-Profile-YAML-v1.0-os.html#_Toc471725209>
> > > > (theory and examples)
> > > > 3.8.1, 3.8.2 (grammar)
> > > > *From Tal's amazing lecture on TOSCA
> > > > :*
> > > > 00:00 until 12:30.
> > > >
> > > > If anyone wishes to:
> > > > * ask questions regarding this feature
> > > > * suggest real-life use cases
> > > > * offer their insight about vague parts of the spec
> > > > * anything else about substitution mapping and service composition
> > > > Then please, feel encouraged to leave your feedback!
> 

Digest for ASF's Slack

[Digest]

Here’s your digest for today! 
#ariatosca
undefined: [UPDATE] started the reviewv with <@U53FAA5HP>  over the 
instantiation module refactory. <@U53P75YTD> I'd be happy if you could review 
it as well :slightly_smiling_face: 
undefined: ok, but only because you begged
undefined: _offended that none here begs me for anything_
undefined: please please please suneel graduate us from incubator
undefined: [UPDATE] started reviewing ARIA-174 (instantiation refactor) with 
<@U546PEP9S>. Continued researching the service composition feature.
undefined: 2 more releases and add atleast 3 more committers - i will do what 
it takes to kick u out of incubator (and out of Apache)
undefined: <@U4SCXA5DL> we'll create our own aria foundation and won't invite 
you! :stuck_out_tongue:
 [UPDATE] implemented the fixes discussed for ARIA-324. can somebody go over 
the new commit for the PR? 


meanwhile i've started working on the cloudify plugin adapter to support our 
new notation.

also, i took the liberty of briefly documenting this notation for the first 
time: 

pip executable expected as part of plugin install.

[D Jayachandran]

Hi,

Am using a Ubuntu version of linux for my development and ARIA does not find 
the correct path of pip during the plugin install.
To be precise this happens when pip freeze is executed.

@staticmethod
def _pip_freeze():
"""Run pip freeze in current environment and return the output"""
bin_dir = 'Scripts' if os.name == 'nt' else 'bin'
pip_path = os.path.join(sys.prefix, bin_dir,
'pip{0}'.format('.exe' if os.name == 'nt' else 
''))
pip_freeze = subprocess.Popen([pip_path, 'freeze'], 
stdout=subprocess.PIPE)
pip_freeze_output, _ = pip_freeze.communicate()
assert not pip_freeze.poll()
return pip_freeze_output

Now the question is why are we executing a pip command directly and not using 
pip as a library.


Regards,
DJ

RE: Inputs and Node object context for python and shell scripts

[D Jayachandran]

Hi Max,

Thanks for the info.  So with this decorator we get only the attributes and 
properties wrapped as dictionary. 
All other node objects like relationships, capabilities and interfaces are as 
wrapped mapped collections.

Regards,
DJ
-Original Message-
From: Maxim Orlov [mailto:ma...@cloudify.co] 
Sent: Tuesday, August 01, 2017 4:24 PM
To: dev@ariatosca.incubator.apache.org
Subject: Re: Inputs and Node object context for python and shell scripts

Sorry for the broken email, it seems my markup translator has some funky 
behavior. The code block is:

from aria import operation

@operation
def samplemethod(ctx=None, **inputs):
  print "ctx -->",ctx
  print "inputs -->",inputs
  ctx.node.attributes['test'] = "abc"


On Tue, Aug 1, 2017 at 1:48 PM Maxim Orlov  wrote:

> Oh, i see. For each method which represents an operation, you should 
> use the @operation decorator. Thus samplemethod would look like this:
>
> from aria import operation
> @operation
>
> def samplemethod(ctx=None, **inputs):
> print "ctx -->",ctx
> print "inputs -->",inputs
> ctx.node.attributes['test'] = "abc"
>
> It is actually this decorator which wraps the attribute model for you.
>
> p.s.
> the ctx comes with its own logger, thus using ctx.logger.info("ctx -->
> {0}".format(ctx)) instead of print "ctx -->", ctx. This will persist 
> your logs, and in case of remote execution, print your logs to the 
> local terminal.
> ​
>
> On Tue, Aug 1, 2017 at 11:30 AM D Jayachandran < 
> d.jayachand...@ericsson.com> wrote:
>
>> Hi Max,
>>
>> I have a service template with just node templates web_app and 
>> database with a depends on Relationship. Both use the same custom 
>> node type derived from "tosca:Root".
>> I just have the create operation defined where the implementation 
>> points to a plugin module. Am trying to set the attribute value in the 
>> plugin.
>> Please find below service template and node types
>>
>> SERVICE TEMPLATE
>>
>> tosca_definitions_version: tosca_simple_yaml_1_0
>>
>> imports:
>>   - types/kubernetes_type.yaml
>>   - aria-1.0
>>
>> topology_template:
>>
>> inputs:
>> web_app_name:
>> type: string
>> value: tosca-webapp
>>
>> web_app_image:
>> type: string
>> value: kuber-master:5000/webwithdbinput
>>
>> web_app_port:
>> type: integer
>> value: 80
>>
>> db_name:
>> type: string
>> value: tosca-database
>>
>> db_image:
>> type: string
>> value: kuber-master:5000/dbforweb
>>
>> db_port:
>> type: integer
>> value: 3306
>>
>>
>> policies:
>>   testplugin:
>> type: aria.Plugin
>> description: policy_description
>> properties:
>> version: 1.2.0
>> enabled: true
>>
>> node_templates:
>> web_app:
>> type: nodes.Container.Application.Kubernetes
>> properties:
>> name: { get_input: web_app_name }
>> image: { get_input: web_app_image }
>> port: { get_input: web_app_port }
>> attributes:
>> test: abc
>> requirements:
>> - dependency:
>>   node: database
>>   relationship:
>>   type: tosca.relationships.DependsOn
>> interfaces:
>> Standard:
>> inputs:
>> name: { get_property: [ web_app, name] }
>> image: { get_property: [ web_app, image] }
>> exposed_port: { get_property: [ web_app, port] }
>> target_host: { get_property: [ database, name] }
>> target_port: { get_property: [ database, port] }
>> isService: true
>> create:
>> inputs:
>> name: { get_property: [ web_app, name] }
>> image: { get_property: [ web_app, image] }
>> exposed_port: { get_property: [ web_app, 
>> port] }
>> target_host: { get_property: [ database, 
>> name] }
>> #target_port: { get_property: [ database, 
>> port] }
>> target_port: 
>> isService: false
>>
>> database:
>> type: nodes.Container.Application.Kubernetes
>> properties:
>> name: { get_input: db_name }
>> image: { get_input: db_image }
>> port: { get_input: db_port }
>> interfaces:
>> Standard:
>> inputs:
>> name: { get_property: [ database, name] }
>> image: { get_property: [ database, image] }
>>

Re: Service Composition / Substitution Mapping

[Tal Liron]

I agree! Luckily metadata exists in the 1.0 spec. :)

http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.0/cos01/TOSCA-Simple-Profile-YAML-v1.0-cos01.html#_Toc379455044

On Tue, Aug 1, 2017 at 7:16 PM, DeWayne Filppi  wrote:

> It occurs that it might be useful to be able to tag service templates with
> arbitrary meta-data.  Perhaps at one level carried forward from a CSAR
> manifest, but also user definable.  This would allow inter-service
> references to be definitive, if desired.  This could be implicitly defined
> as a capability by the orchestrator, but some kind of special requirement
> type(s) would be needed to utilize it.  This way, external repos could be
> used safely and directly without the separate load step.
>
> On Tue, Aug 1, 2017 at 12:43 PM, Tal Liron  wrote:
>
> > Thanks for the kudos. :)
> >
> > This topic was discussed on this list a while ago. It's indeed tricky to
> > get right, because TOSCA leaves a lot of room for the orchestrator to
> > implement.
> >
> > I'm thinking of it working something like this:
> >
> > 1. The reqs-and-caps engine by default will always look for satisfiable
> > capabilities within the currently instantiated service. HOWEVER, if such
> a
> > capability is not present, the option is there to look for another
> > instantiated service that exposes the capabilities in substitution
> > mappings.
> >
> > 2. If we DON'T have another instantiated service, but DO have a service
> > template that could fit the bill, perhaps we need to instantiate that
> other
> > service first. One obvious option is to do this automatically. But I feel
> > like this can create unforeseen consequences -- for example, some dummy
> > test template that someone happened to have in the database might get
> > instantiated by mistake. Also, it might need to trigger multiple install
> > workflows at once... a big mess. So I suggest that instead we provide a
> > very detailed validation error here saying that the requirement cannot be
> > satisfied, HOWEVER there exist service templates A, B, and C that can
> > substitute for us, so maybe the nice user would like to instantiate them
> > first? This seems very reasonable to me.
> >
> > 3. If indeed another service satisfies this, a special node is added to
> the
> > current service (with the correct type -- but without a service template
> > foreign key), which serves as a proxy of the other service template. I'm
> > not sure how we would mark this exactly. We can't use the service_fk
> field,
> > because it's still in our current service. So perhaps there's need of a
> new
> > fk field, maybe substituted_service_fk?
> >
> > The above might be "sensible defaults," but it seems to me that users
> > really need control over this. So I propose to add a new aria.Composition
> > policy that would let you provide hints for this mechanism. For example,
> > you might want to "filter" the target service by service template name
> and
> > even by metadata in the service template. For example, you might want to
> > require version 1.2.2 of a specific service, no less.
> >
> > Those are some quick thoughts. Exactly how such a policy would look with
> > require more thought...
> >
> >
> > On Tue, Aug 1, 2017 at 2:20 PM, Avia Efrat  wrote:
> >
> > > Hello all,
> > >
> > > I'm starting to work on a full implementation of substitution_mapping,
> > > which will lead to the ability of service composition.
> > >
> > > For those unacquainted with substitution mapping, here are some quick
> > > resources:
> > > *From the spec
> > >  > > YAML/v1.0/TOSCA-Simple-Profile-YAML-v1.0.html>,
> > > sections:*
> > > 2.10
> > >  > > YAML/v1.0/os/TOSCA-Simple-Profile-YAML-v1.0-os.html#_Toc471725208>,
> > > 2.11
> > >  > > YAML/v1.0/os/TOSCA-Simple-Profile-YAML-v1.0-os.html#_Toc471725209>
> > > (theory and examples)
> > > 3.8.1, 3.8.2 (grammar)
> > > *From Tal's amazing lecture on TOSCA
> > > :*
> > > 00:00 until 12:30.
> > >
> > > If anyone wishes to:
> > > * ask questions regarding this feature
> > > * suggest real-life use cases
> > > * offer their insight about vague parts of the spec
> > > * anything else about substitution mapping and service composition
> > > Then please, feel encouraged to leave your feedback!
> > >
> >
>