Re: error after adding "userdata" or "user_data" to plugin.yaml

[DeWayne Filppi]

I have no idea if its a bug.  Just seeing if it looked familiar to anyone.
Thought that error message might ring a bell.

On Sep 5, 2017 10:33 PM, "Tal Liron"  wrote:

> Are you reporting a bug here? It would really help to get a complete
> example with instructions to reproduce.
>
> On Tue, Sep 5, 2017 at 7:07 PM, DeWayne Filppi 
> wrote:
>
> > After adding the highlighted content to the ARIA openstack plugin.yaml,
> >
> >   aria.openstack.datatypes.Server:
> > description: >-
> >   openstack Server args.
> > properties:
> >   server:
> > type: map
> > required: false
> > entry_schema:
> >   type: string
> >
> > I get the following error trying to use user_data in a template:
> >
> > RuntimeError: Re-definition of yaml-1.1 in specification_url
> >
> > Usage looks like:
> >
> >   openstack_config: { get_input: openstack_config }
> >   args:
> > server:
> >   user_data: |
> > #!/bin/sh
> > echo TEST > /tmp/out
> >
>

Re: error after adding "userdata" or "user_data" to plugin.yaml

[Tal Liron]

Are you reporting a bug here? It would really help to get a complete
example with instructions to reproduce.

On Tue, Sep 5, 2017 at 7:07 PM, DeWayne Filppi  wrote:

> After adding the highlighted content to the ARIA openstack plugin.yaml,
>
>   aria.openstack.datatypes.Server:
> description: >-
>   openstack Server args.
> properties:
>   server:
> type: map
> required: false
> entry_schema:
>   type: string
>
> I get the following error trying to use user_data in a template:
>
> RuntimeError: Re-definition of yaml-1.1 in specification_url
>
> Usage looks like:
>
>   openstack_config: { get_input: openstack_config }
>   args:
> server:
>   user_data: |
> #!/bin/sh
> echo TEST > /tmp/out
>

Podling Report Reminder - September 2017

[johndament]

Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 20 September 2017, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, September 06).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.
*   How does the podling rate their own maturity.

This should be appended to the Incubator Wiki page at:

https://wiki.apache.org/incubator/September2017

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

error after adding "userdata" or "user_data" to plugin.yaml

[DeWayne Filppi]

After adding the highlighted content to the ARIA openstack plugin.yaml,

  aria.openstack.datatypes.Server:
description: >-
  openstack Server args.
properties:
  server:
type: map
required: false
entry_schema:
  type: string

I get the following error trying to use user_data in a template:

RuntimeError: Re-definition of yaml-1.1 in specification_url

Usage looks like:

  openstack_config: { get_input: openstack_config }
  args:
server:
  user_data: |
#!/bin/sh
echo TEST > /tmp/out

Re: openstack plugin

[DeWayne Filppi]

Yes, I believe you've got the idea.  The original "server" arg is defined
as a map/dict.  TOSCA of course requires something somewhat more explicit.
I'm working around it by adding the following to Server datatype:

  aria.openstack.datatypes.Server:
description: >-
  openstack Server args.
properties:
  server:
type: map
required: false
entry_schema:
  type: string
  security_groups:
type: list
entry_schema: string
required: false
  availability_zone:
type: string
required: false

I have my own copy of the plugin.yaml file that I'm making changes to so
things will work for me, so this is fine.  I think the PR needs to go
beyond this, and ensure that all potential parameters are defined ( or at
least the parameters that the Openstack plugin recognizes).



On Sun, Sep 3, 2017 at 12:26 AM, Ran Ziv  wrote:

> Could you link to the relevant place?
> Do you mean that in the original plugin.yaml you could set the userdata
> field (under properties? under the "server" property? args?) but only due
> to it not having had to be declared explicitly?
> There's a good chance that if this were possible than it simply got omitted
> during translation by mistake.
> Feel free to create a PR and add that.
>
> As a workaround, I assume you can simply inherit the type.
>
>
> On Sat, Sep 2, 2017 at 7:02 PM, DeWayne Filppi 
> wrote:
>
> > The openstack plugin.yaml replacement doesn't permit userdata.  Unlike
> the
> > regular plugin.yaml, the args contents are specifically defined, and that
> > definition excludes userdata.  Is this a problem or is there some kind of
> > workaround?
> >
>

Re: Service Composition / Substitution Mapping

[Tal Liron]

On Sun, Sep 3, 2017 at 4:06 AM, Ran Ziv  wrote:

> I too agree with the notion that picking a substituting service template
> explicitly is against the spirit of this feature, for the reasons Avia has
> explained.
> If a specific template is needed, why not import it directly, or require a
> more specific capability offered by the substituting service template?
>

Just like you can specify a specific node template for a requirement, I
believe it can be useful and often necessary to specify a substituted
service template.

Let me try an example:

topology_template:
  node_templates:
my_app:
  type: WebApp
  requirements:
host:
  capability: tosca.capabilities.Container
  node: my_host # explicit node template
db:
  capability: MongoDb
my_host:
  type: VirtualMachine

  policies:
substitution:
  type: aria.Substitution
  targets: [ my_app ]
  properties:
requirements: # map of strings to requirement policy infos
  db:
service_template: /opt/templates/MongoDbCluster.csar # explicit
service template

Re: Service Composition / Substitution Mapping

[Tal Liron]

On Tue, Sep 5, 2017 at 5:33 AM, D Jayachandran 
wrote:

>  Hence I wanted to know if there is any possibility of identifying an
> abstract type (during parsing )so that we need not import or define the
> custom node type in my service template.
>

In my opinion, the import of the abstract type is necessary. TOSCA offers
no alternative, and I don't see how ARIA would be able to guess that this
type is abstract nor have any notion where to find the type. I think that
such usability problems we find in the mechanism need to be addressed at
OASIS. Perhaps we can make TOSCA 1.2 easier to use with substitution
mapping. For now, though, I think ARIA should adhere to the TOSCA 1.0
standard as closely as possible.

Re: Unique identification of an instance element across services

[Tal Liron]

We already have utility code to generate all kinds of UUIDs, so it's
trivial to make the change. I guess it's just a matter of making a
decision...

On Tue, Sep 5, 2017 at 3:57 AM, Vaishnavi K.R 
wrote:

>
> Hi,
>
> I agree that with the CLI based usage in ARIA, the requirement of the UUID
> based identification of the node and service instance elements is an
> overhead.
>
> From the discussions so far, it seems like UUID is important in handling
> the multi-user and multi-tenant scenarios.
>
> Do you have any update on when UUID will be considered in the roadmap?
> If its not too far, we would like to make our contribution to ARIA on UUID.
>
> Looking forward to your response.
>
>
> Thanks,
>
> /Vaish
>
> 
> From: Avia Efrat 
> Sent: Sunday, July 30, 2017 10:35:45 PM
> To: dev@ariatosca.incubator.apache.org
> Subject: Re: Unique identification of an instance element across services
>
> First, good arguments from both 'sides'.
>
> I am for at least adding a uuid as an option, as ARIA is intended to be
> used at scale as well.
> But currently, I am for the simple ids to be used as default (and not
> uuids). Like it or not, right now ARIA is more at a 'TOSCA playground'
> stage, and I think that's perfectly fine =)
> And at this stage, I think simple ids will be better, as they easier to use
> via the CLI, but more importantly, don't clog the logs with long
> meaningless strings. As ARIA matures, we could switch the default to UUIDs.
>
> And BTW, as our log format is configurable, there could be other ways than
> UUIDs to distinguish between nodes with the 'same id' in a central logging
> system, e.g using the user name as another indicator.
>
> On Thu, Jul 27, 2017 at 10:20 AM, Vaishnavi K.R <
> vaishnavi@ericsson.com>
> wrote:
>
> > Hi,
> >
> >
> > Thanks for the active discussion.
> >
> >
> > Having UUID at the node instance level will just make the nodes unique.
> >
> > And these names will not be used by the user directly as no operations
> are
> > happening on the node instance name.
> >
> > But at the service template and the service level, UUID will be of great
> > help considering the multi user and multi tenancy situations.
> >
> >
> > So in a large scale perspective, the node names and the service template
> > names have high probability of being same.
> >
> > When these enter into the automated world, it will create more problem
> > when name conflicts occur and its adds overhead to make changes every
> time
> > to overcome the conflict.
> >
> >
> > UUID at service template and the service level: will be of much use in
> the
> > above scenario and operations by user on these are less
> >
> > UUID at node instance level: makes the node much unique and no operation
> > happens on it
> >
> >
> > Thanks,
> >
> > /Vaish
> >
> > 
> > From: Tal Liron 
> > Sent: Wednesday, July 26, 2017 8:48:40 PM
> > To: dev@ariatosca.incubator.apache.org
> > Subject: Re: Unique identification of an instance element across services
> >
> > I just don't see users having to deal much with node IDs outside of
> simple
> > hello-world style tutorials, and I'd hate for the first impressions that
> > users get out of ARIA is that it's just a playground for TOSCA. It should
> > be ready out-of-the-box for the real world.
> >
> > On Wed, Jul 26, 2017 at 9:13 AM, DeWayne Filppi 
> > wrote:
> >
> > > Such is their strength.  I'm just advocating using them as a last
> resort
> > > because they are user unfriendly and gigantic.
> > >
> > > On Tue, Jul 25, 2017 at 12:55 PM, Tal Liron  wrote:
> > >
> > > > Let's consider a mass deployment: thousands of service instances of
> the
> > > > same service template, created by many different users with their own
> > > ARIA
> > > > installations (and databases). In that case, assuming we use
> sequential
> > > > IDs, you would have the same node ID appear many times. You would
> have
> > to
> > > > identify it via the particular user and service instance. If you're
> > > > centralizing logs, this can quickly be cumbersome. A UUID will
> identify
> > > it
> > > > globally and avoid any confusion.
> > > >
> > > > I think the default should be something that avoids such problems.
> For
> > > > users who insist on shorter IDs, we can allow them to configure it.
> > > >
> > > > On Tue, Jul 25, 2017 at 2:42 PM, DeWayne Filppi  >
> > > > wrote:
> > > >
> > > > > True uuids are seductive, because of their simplicity.  But they
> are
> > > > huge,
> > > > > overkill, and meaningless.  Imho a structured id is superior if it
> > can
> > > be
> > > > > made to work without a global locking scheme.
> > > > >
> > > > > - DeWayne
> > > > >
> > > > > On Jul 25, 2017 12:11 PM, "Tal Liron"  wrote:
> > > > >
> > > > > > It's not an issue of thread safety -- it could be entirely
> > different
> > > > > 

RE: Service Composition / Substitution Mapping

[D Jayachandran]

Hi All,

Thanks for the comments on handling substitution when multiple service 
templates matches for the abstract node type.

My 2nd point was on importing the non-normative node types in the top level 
service template. Let me explain this with an example.

1)  Normative type (tosca.nodes.Database) as abstract node
  db:
  # This node is abstract (no Deploment or Implemenation artifacts on 
create)
  # and can be substituted with a topology provided by another template
  # that exports a Database type’s capabilities.
  type: tosca.nodes.Database
  properties:
user: my_db_user
password: secret
name: my_db_name

With this example, the abstract node type is a normative type from the simple 
YAML 1.0 profile. Hence with our proposed approach we will go through the 
available service templates and find the required substitution.

2) Non-normative type (myapp.nodes.Database)
If I replace the above example with the non-normative type " 
myapp.nodes.Database" , we need to import this type in our service template to 
have the parsing complete. 

import types/my_custom.yaml
 
node_templates: 
  db:
  # This node is abstract (no Deploment or Implemenation artifacts on 
create)
  # and can be substituted with a topology provided by another template
  # that exports a Database type’s capabilities.
  type: myapp.nodes.Database
  properties:
user: my_db_user
password: secret
name: my_db_name

Here "db" is an abstract node of type  "myapp.nodes.Database" which needs to be 
substituted from other service templates. Though it will be substituted we need 
to have this type "myapp.nodes.Database" defined in my top level service 
template or imported for the parsing to be sucessfull.  Hence I wanted to know 
if there is any possibility of identifying an abstract type (during parsing )so 
that we need not import or define the custom node type in my service template. 

Regards,
DJ
-Original Message-
From: Ran Ziv [mailto:r...@cloudify.co] 
Sent: Sunday, September 03, 2017 2:36 PM
To: dev@ariatosca.incubator.apache.org
Subject: Re: Service Composition / Substitution Mapping

I'm not sure I've followed why a policy would be necessary.

I too agree with the notion that picking a substituting service template 
explicitly is against the spirit of this feature, for the reasons Avia has 
explained.
If a specific template is needed, why not import it directly, or require a more 
specific capability offered by the substituting service template?


On Thu, Aug 31, 2017 at 7:09 PM, Tal Liron  wrote:

> Regarding #1, I agree with Avia here. Substitution mapping is intended 
> as an extension of requirements-matching-capabilities. As you know, a 
> requirement can be as minimal as just specifying a capability type, 
> but could optionally additionally specify a node and even include a 
> node filter with more constraints. Unfortunately the substitution 
> mapping grammar is still not very sophisticated in TOSCA 1.0, and 
> while you can require a node, you can't specifically require it to be 
> a substituted service template. That's why I'm thinking we may need to 
> introduce a policy to fill in this gap for now. But indeed, if there 
> is no such policy, available service templates could be tried 
> arbitrarily one at a time until one satisfies the requirement (as Avia 
> points out, this would require the composite service to be *entirely* 
> successfully satisfied).
>
> On Thu, Aug 31, 2017 at 10:23 AM, Avia Efrat  wrote:
>
> > 1) Substitutable nodes (abstract nodes) are defined by their type. 
> > And a node type should represent specific node/subsystem characteristics.
> Having
> > two node types ( with the same name) that represent different 
> > logical entities seems odd, and indicates a possible bug in the 
> > design. That is another reason why using so-called namespaces while 
> > creating type hierarchies is a good practice.
> >
> > You could argue that if you have multiple sources for templates (e.g.
> your
> > own storage and a external catalog), there could be unexpected name 
> > collisions of node types names. Having said that, we need to 
> > remember the entire process of the substitution mapping.
> >
> > First, as you said, we are trying to locate a service template that 
> > has a substitution mapping definition of the same type of the 
> > abstract node in the top level template. But choosing the service 
> > template is only the
> first
> > step. The second step is instantiating the top level template along 
> > with the substitution template. During the instantiation, 
> > requirements and capabilities are matched. If not all requirements 
> > are satisfied (considering both top level and substituting 
> > template), the substitution mapping is not considered successful, and the 
> > instantiation fails.
> >
> > If that is the case, it makes sense to go ahead and 

Re: Unique identification of an instance element across services

[Vaishnavi K . R]

Hi,

I agree that with the CLI based usage in ARIA, the requirement of the UUID 
based identification of the node and service instance elements is an overhead.

>From the discussions so far, it seems like UUID is important in handling the 
>multi-user and multi-tenant scenarios.

Do you have any update on when UUID will be considered in the roadmap?
If its not too far, we would like to make our contribution to ARIA on UUID.

Looking forward to your response.


Thanks,

/Vaish


From: Avia Efrat 
Sent: Sunday, July 30, 2017 10:35:45 PM
To: dev@ariatosca.incubator.apache.org
Subject: Re: Unique identification of an instance element across services

First, good arguments from both 'sides'.

I am for at least adding a uuid as an option, as ARIA is intended to be
used at scale as well.
But currently, I am for the simple ids to be used as default (and not
uuids). Like it or not, right now ARIA is more at a 'TOSCA playground'
stage, and I think that's perfectly fine =)
And at this stage, I think simple ids will be better, as they easier to use
via the CLI, but more importantly, don't clog the logs with long
meaningless strings. As ARIA matures, we could switch the default to UUIDs.

And BTW, as our log format is configurable, there could be other ways than
UUIDs to distinguish between nodes with the 'same id' in a central logging
system, e.g using the user name as another indicator.

On Thu, Jul 27, 2017 at 10:20 AM, Vaishnavi K.R 
wrote:

> Hi,
>
>
> Thanks for the active discussion.
>
>
> Having UUID at the node instance level will just make the nodes unique.
>
> And these names will not be used by the user directly as no operations are
> happening on the node instance name.
>
> But at the service template and the service level, UUID will be of great
> help considering the multi user and multi tenancy situations.
>
>
> So in a large scale perspective, the node names and the service template
> names have high probability of being same.
>
> When these enter into the automated world, it will create more problem
> when name conflicts occur and its adds overhead to make changes every time
> to overcome the conflict.
>
>
> UUID at service template and the service level: will be of much use in the
> above scenario and operations by user on these are less
>
> UUID at node instance level: makes the node much unique and no operation
> happens on it
>
>
> Thanks,
>
> /Vaish
>
> 
> From: Tal Liron 
> Sent: Wednesday, July 26, 2017 8:48:40 PM
> To: dev@ariatosca.incubator.apache.org
> Subject: Re: Unique identification of an instance element across services
>
> I just don't see users having to deal much with node IDs outside of simple
> hello-world style tutorials, and I'd hate for the first impressions that
> users get out of ARIA is that it's just a playground for TOSCA. It should
> be ready out-of-the-box for the real world.
>
> On Wed, Jul 26, 2017 at 9:13 AM, DeWayne Filppi 
> wrote:
>
> > Such is their strength.  I'm just advocating using them as a last resort
> > because they are user unfriendly and gigantic.
> >
> > On Tue, Jul 25, 2017 at 12:55 PM, Tal Liron  wrote:
> >
> > > Let's consider a mass deployment: thousands of service instances of the
> > > same service template, created by many different users with their own
> > ARIA
> > > installations (and databases). In that case, assuming we use sequential
> > > IDs, you would have the same node ID appear many times. You would have
> to
> > > identify it via the particular user and service instance. If you're
> > > centralizing logs, this can quickly be cumbersome. A UUID will identify
> > it
> > > globally and avoid any confusion.
> > >
> > > I think the default should be something that avoids such problems. For
> > > users who insist on shorter IDs, we can allow them to configure it.
> > >
> > > On Tue, Jul 25, 2017 at 2:42 PM, DeWayne Filppi 
> > > wrote:
> > >
> > > > True uuids are seductive, because of their simplicity.  But they are
> > > huge,
> > > > overkill, and meaningless.  Imho a structured id is superior if it
> can
> > be
> > > > made to work without a global locking scheme.
> > > >
> > > > - DeWayne
> > > >
> > > > On Jul 25, 2017 12:11 PM, "Tal Liron"  wrote:
> > > >
> > > > > It's not an issue of thread safety -- it could be entirely
> different
> > > > > processes, on different machines, accessing the same db. It can be
> > > solved
> > > > > via a SQL transaction, but I feel the whole issue can be avoided by
> > > using
> > > > > UUIDs.
> > > > >
> > > > > Using the CLI to access specific nodes is not something I see
> > > happening a
> > > > > lot outside of debugging. And when you do debug, you'll probably be
> > > > copying
> > > > > and pasting a node ID from the logs, so shorter names do not add
> much
> > > > ease
> > > > > of use.
>